| Title: | Statistical Consistency Checker for Published Research Results |
|---|---|
| Description: | A conservative, assumption-aware statistical consistency checker for already-extracted research-results text. Parses test statistics, effect sizes, and confidence intervals across multiple citation styles including American Psychological Association (APA), Harvard, Frontiers, PLOS ONE, Scientific Reports, Nature Human Behaviour, PeerJ, eLife, PNAS, and others. Recomputes effect sizes using all plausible variants when design is ambiguous, and validates internal consistency. Supports t-tests, F-tests/ANOVA, correlations, chi-square, z-tests, regression, and nonparametric tests. Explicitly tracks all assumptions and uncertainty in output. Detects decision errors (significance reversals) similar to 'statcheck'. From v0.4.0 file extraction is no longer part of the package — pair with an external extractor (e.g., docpluck at <https://docpluck.app>) and pass the resulting text to check_text(). Note: this package is under active development and results should be independently verified. Use is at the sole responsibility of the user. Contributions and verification reports are welcome. |
| Authors: | Gilad Feldman [aut, cre] (ORCID: <https://orcid.org/0000-0003-2812-6599>) |
| Maintainer: | Gilad Feldman <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.6.2 |
| Built: | 2026-05-25 17:09:37 UTC |
| Source: | https://github.com/giladfeldman/escicheck |
This file defines the effectcheck S3 class and its associated methods for printing, summarizing, and plotting results.
.effectcheck_version().effectcheck_version()
Preserves effectcheck class when subsetting.
## S3 method for class 'effectcheck' x[...]## S3 method for class 'effectcheck' x[...]
x |
An effectcheck object |
... |
Subsetting arguments |
An effectcheck object
res <- check_text("t(28) = 2.21, p = .035. F(1, 50) = 4.03, p = .049") res[1, ]res <- check_text("t(28) = 2.21, p = .035. F(1, 50) = 4.03, p = .049") res[1, ]
Parses APA-style statistical results from text and checks for consistency between reported and computed values. Uses type-matched comparison to ensure reported effect sizes are compared against the same type of computed values.
check_text( text, stats = c("t", "F", "r", "chisq", "z", "U", "W", "H", "regression", "spearman", "kendall", "kendall_w", "dscf", "cochran_q", "RR", "rdpct", "md_hl", "binomial"), ci_level = 0.95, alpha = 0.05, one_tailed = FALSE, paired_r_grid = c(seq(0.1, 0.9, by = 0.1), 0.95), assume_equal_ns_when_missing = TRUE, ci_method_phi = "bonett_price", ci_method_V = "bonett_price", tol_effect = list(d = 0.02, r = 0.005, phi = 0.02, V = 0.02), tol_ci = 0.02, tol_p = 0.001, messages = FALSE, max_text_length = 10^7, max_stats_per_text = 10000, cross_type_action = "NOTE", ci_affects_status = TRUE, plausibility_filter = TRUE, sign_sensitive = FALSE, method_context_action = "NOTE", design_ambiguous_action = "WARN", unknown_groups_action = "WARN", min_confidence = 0L )check_text( text, stats = c("t", "F", "r", "chisq", "z", "U", "W", "H", "regression", "spearman", "kendall", "kendall_w", "dscf", "cochran_q", "RR", "rdpct", "md_hl", "binomial"), ci_level = 0.95, alpha = 0.05, one_tailed = FALSE, paired_r_grid = c(seq(0.1, 0.9, by = 0.1), 0.95), assume_equal_ns_when_missing = TRUE, ci_method_phi = "bonett_price", ci_method_V = "bonett_price", tol_effect = list(d = 0.02, r = 0.005, phi = 0.02, V = 0.02), tol_ci = 0.02, tol_p = 0.001, messages = FALSE, max_text_length = 10^7, max_stats_per_text = 10000, cross_type_action = "NOTE", ci_affects_status = TRUE, plausibility_filter = TRUE, sign_sensitive = FALSE, method_context_action = "NOTE", design_ambiguous_action = "WARN", unknown_groups_action = "WARN", min_confidence = 0L )
text |
Character vector of text to check |
stats |
Character vector of test types to check (default: all supported types) |
ci_level |
Default confidence interval level (default 0.95) |
alpha |
Significance threshold for decision error detection (default 0.05) |
one_tailed |
Logical, assume one-tailed tests (default FALSE) |
paired_r_grid |
Numeric vector of correlation values for paired t-test grid search |
assume_equal_ns_when_missing |
Logical, assume equal group sizes when missing (default TRUE) |
ci_method_phi |
CI method for phi coefficient (default "bonett_price") |
ci_method_V |
CI method for Cramer's V (default "bonett_price") |
tol_effect |
List of tolerances for effect sizes by type |
tol_ci |
Tolerance for CI bounds (default 0.02) |
tol_p |
Tolerance for p-values (default 0.001) |
messages |
Logical, show progress messages (default FALSE) |
max_text_length |
Maximum total text length in characters (default 10^7) |
max_stats_per_text |
Maximum number of stats to process per text (default 10000) |
cross_type_action |
Action when cross-type match found ("NOTE", "WARN", or "ERROR"; default "NOTE") |
ci_affects_status |
Whether CI mismatches affect status (default TRUE) |
plausibility_filter |
Whether to apply plausibility bounds filter (default TRUE) |
sign_sensitive |
Whether sign differences affect status (default FALSE) |
method_context_action |
Action when method context detected in chunk ("NOTE", "WARN", or "SKIP"; default "NOTE") |
design_ambiguous_action |
Action when design-ambiguous t-test (or F(1,df)) effect size ERROR occurs ("WARN", "NOTE", or "ERROR"; default "WARN") |
unknown_groups_action |
Action when d/g ERROR occurs with unknown group sizes n1/n2 ("WARN", "NOTE", or "ERROR"; default "WARN") |
min_confidence |
Minimum confidence score (0-10) for results to be included in output (default 0) |
An effectcheck S3 object whose results tibble carries the
parsed and recomputed statistics. Notable output columns:
design_ambiguousLogical. TRUE when the row's matching
is design-uncertain. INTENTIONALLY BROAD – see ambiguity_reason
for the specific category (one of two: structural-design for
a t / F(1,df) / z that reports d or g and produced BOTH paired and
independent variant families; cross-family for a reported ES
type that has no same-type variants in the computed-variants set,
e.g. a Cohen's d reported on an F(2,df) omnibus). Equal to
ambiguity_level != "clear"; the flag never under-reports the
row's uncertainty.
ambiguity_levelCharacter. "clear",
"ambiguous", or "highly_ambiguous". The category-A
cases tend to land on "ambiguous"; the category-B cases land
on "highly_ambiguous".
ambiguity_reasonCharacter. Human-readable explanation
of the ambiguity. Since v0.5.11, a stable bracket-tagged category
suffix is appended when applicable: "[category: structural-design]"
or "[category: cross-family]". Consumers can grep for the tag
to programmatically split the two semantics without parsing English.
matched_variantThe computed variant matched against
the reported ES. A cross-family fallback row will name a variant
from a different family than the reported ES type (e.g.
matched_variant="eta" when effect_reported_name="d").
Plus all other columns: location, raw_text, test
identification (test_type, chisq_subtype, df1,
df2, stat_value, N), p-values (p_reported,
p_computed, decision_error), effect-size family columns
(d_ind, dz, g_ind, eta2, partial_eta2,
omega2, ...), CI metadata (ci_reported, ci_expected,
ci_width_ratio, ci_level_source, ...), and status
(status, check_type, check_scope,
extraction_suspect, design_inferred,
uncertainty_level, uncertainty_reasons, ...).
result <- check_text("t(28) = 2.21, p = .035, d = 0.80") print(result) summary(result)result <- check_text("t(28) = 2.21, p = .035, d = 0.80") print(result) summary(result)
Creates a comparison table showing the reported value against all computed variants.
compare_to_variants(x, row_index = 1)compare_to_variants(x, row_index = 1)
x |
An effectcheck object |
row_index |
The row index |
A data frame with variant comparisons
res <- check_text("t(28) = 2.21, p = .035, d = 0.80") compare_to_variants(res, 1)res <- check_text("t(28) = 2.21, p = .035, d = 0.80") compare_to_variants(res, 1)
Runs both effectcheck and statcheck on the same text and returns a merged comparison tibble.
compare_with_statcheck(text, ...)compare_with_statcheck(text, ...)
text |
Character string containing APA-formatted statistics |
... |
Additional arguments passed to check_text() |
A tibble with source column ("both", "effectcheck_only", "statcheck_only")
comp <- compare_with_statcheck("t(28) = 2.21, p = .035, d = 0.80") print(comp)comp <- compare_with_statcheck("t(28) = 2.21, p = .035, d = 0.80") print(comp)
Provides counts of statistics grouped by various categories.
count_by(x, by = c("status", "test_type", "uncertainty", "design", "source"))count_by(x, by = c("status", "test_type", "uncertainty", "design", "source"))
x |
An effectcheck object |
by |
Character, grouping variable: "status", "test_type", "uncertainty", "design", or "source" |
A data frame with counts
results <- check_text("t(28) = 2.21, p = .035. F(1, 50) = 4.03, p = .049") count_by(results, "status") count_by(results, "test_type")results <- check_text("t(28) = 2.21, p = .035. F(1, 50) = 4.03, p = .049") count_by(results, "status") count_by(results, "test_type")
Functions for filtering and identifying problematic results in effectcheck output. Identify problematic results
ec_identify( x, what = c("errors", "warnings", "decision_errors", "high_uncertainty", "insufficient", "all_problems"), ... )ec_identify( x, what = c("errors", "warnings", "decision_errors", "high_uncertainty", "insufficient", "all_problems"), ... )
x |
An effectcheck object |
what |
Character vector specifying what to identify:
|
... |
Additional arguments (ignored) |
Filters effectcheck results to show only problematic cases based on specified criteria.
An effectcheck object containing only the identified results
results <- check_text("t(28) = 2.21, p = .035, d = 0.80") errors <- ec_identify(results, "errors")results <- check_text("t(28) = 2.21, p = .035, d = 0.80") errors <- ec_identify(results, "errors")
All file-input functions in this file became .Defunct() in
effectcheck 0.4.0. ESCImate delegates document extraction to
docpluck; pass the resulting
text to check_text() for analysis.
Migration:
## Before (errors in 0.4.0):
results <- effectcheck::checkPDFdir("path/to/pdfs/")
## After:
library(httr2)
pdfs <- list.files("path/to/pdfs/", pattern = "\\.pdf$", full.names = TRUE)
results <- purrr::map_dfr(pdfs, function(p) {
resp <- request("https://docpluck.app/api/extract") |>
req_headers(Authorization = paste("Bearer", Sys.getenv("DOCPLUCK_API_KEY"))) |>
req_url_query(normalize = "academic", quality = "true") |>
req_body_multipart(file = curl::form_file(p)) |>
req_perform()
dplyr::mutate(check_text(resp_body_json(resp)$text), source = basename(p))
})
Exports check results to CSV format with proper handling of special characters and NA values.
export_csv(res, out, na = "", row.names = FALSE)export_csv(res, out, na = "", row.names = FALSE)
res |
tibble returned by check_text() / check_files() |
out |
output file path (csv) |
na |
string to use for NA values (default: "") |
row.names |
logical, include row names (default: FALSE) |
Invisible path to the generated CSV file.
res <- check_text("t(28) = 2.21, p = .035, d = 0.80") export_csv(res, out = tempfile(fileext = ".csv"))res <- check_text("t(28) = 2.21, p = .035, d = 0.80") export_csv(res, out = tempfile(fileext = ".csv"))
Exports check results to JSON format with structured metadata.
export_json(res, out, pretty = TRUE)export_json(res, out, pretty = TRUE)
res |
tibble returned by check_text() / check_files() |
out |
output file path (json) |
pretty |
logical, pretty-print JSON (default: TRUE) |
Invisible path to the generated JSON file.
res <- check_text("t(28) = 2.21, p = .035, d = 0.80") export_json(res, out = tempfile(fileext = ".json"))res <- check_text("t(28) = 2.21, p = .035, d = 0.80") export_json(res, out = tempfile(fileext = ".json"))
Filters effectcheck results by the magnitude of effect size discrepancy.
filter_by_delta(x, min_delta = 0, max_delta = Inf)filter_by_delta(x, min_delta = 0, max_delta = Inf)
x |
An effectcheck object |
min_delta |
Minimum absolute delta to include (default 0) |
max_delta |
Maximum absolute delta to include (default Inf) |
An effectcheck object containing only results within the delta range
results <- check_text("t(28) = 2.21, p = .035, d = 0.80") filter_by_delta(results, min_delta = 0.1)results <- check_text("t(28) = 2.21, p = .035, d = 0.80") filter_by_delta(results, min_delta = 0.1)
Filters effectcheck results to show only results from specific files.
filter_by_source(x, files, pattern = FALSE)filter_by_source(x, files, pattern = FALSE)
x |
An effectcheck object |
files |
Character vector of file names or patterns to include |
pattern |
Logical, if TRUE treat files as regex patterns (default FALSE) |
An effectcheck object containing only results from specified files
results <- check_text("t(28) = 2.21, p = .035, d = 0.80") filter_by_source(results, "text_input")results <- check_text("t(28) = 2.21, p = .035, d = 0.80") filter_by_source(results, "text_input")
Filters effectcheck results to show only specific test types.
filter_by_test_type(x, types)filter_by_test_type(x, types)
x |
An effectcheck object |
types |
Character vector of test types to include (e.g., "t", "F", "r", "chisq", "z") |
An effectcheck object containing only the specified test types
results <- check_text("t(28) = 2.21, p = .035. F(1, 50) = 4.03, p = .049") filter_by_test_type(results, "t")results <- check_text("t(28) = 2.21, p = .035. F(1, 50) = 4.03, p = .049") filter_by_test_type(results, "t")
Filters effectcheck results by uncertainty level.
filter_by_uncertainty(x, levels)filter_by_uncertainty(x, levels)
x |
An effectcheck object |
levels |
Character vector of uncertainty levels to include ("low", "medium", "high") |
An effectcheck object containing only the specified uncertainty levels
results <- check_text("t(28) = 2.21, p = .035, d = 0.80") filter_by_uncertainty(results, "high")results <- check_text("t(28) = 2.21, p = .035, d = 0.80") filter_by_uncertainty(results, "high")
Creates a formatted string representation of variants for a row.
format_variants(x, row_index = 1, include_alternatives = TRUE)format_variants(x, row_index = 1, include_alternatives = TRUE)
x |
An effectcheck object |
row_index |
The row index |
include_alternatives |
Whether to include alternative suggestions |
A character string with formatted variant information
res <- check_text("t(28) = 2.21, p = .035, d = 0.80") cat(format_variants(res, 1))res <- check_text("t(28) = 2.21, p = .035, d = 0.80") cat(format_variants(res, 1))
Creates a self-contained HTML report with executive summary, color-coded results table, expandable details, reproducible R code, and footer stamp.
generate_report( res, out, format = "html", title = "EffectCheck Report", author = NULL, source_name = NULL, include_repro_code = TRUE, style = "beginner" )generate_report( res, out, format = "html", title = "EffectCheck Report", author = NULL, source_name = NULL, include_repro_code = TRUE, style = "beginner" )
res |
tibble returned by check_text() / check_files() |
out |
output file path (html) |
format |
Output format: "html" (default) or "pdf" (requires rmarkdown) |
title |
Report title (default: "EffectCheck Report") |
author |
Author name (optional) |
source_name |
Source file name (optional) |
include_repro_code |
Logical, include reproducible R code section (default TRUE) |
style |
Report style: "beginner" for plain English narrative (default), "expert" for the traditional technical table format |
Invisible path to the generated report file
res <- check_text("t(28) = 2.21, p = .035, d = 0.80") generate_report(res, out = tempfile(fileext = ".html"))res <- check_text("t(28) = 2.21, p = .035, d = 0.80") generate_report(res, out = tempfile(fileext = ".html"))
Get alternative suggestions for a row
get_alternatives(x, row_index = 1)get_alternatives(x, row_index = 1)
x |
An effectcheck object |
row_index |
The row index |
A list of alternative effect size suggestions
res <- check_text("t(28) = 2.21, p = .035, d = 0.80") get_alternatives(res, 1)res <- check_text("t(28) = 2.21, p = .035, d = 0.80") get_alternatives(res, 1)
Extracts results where the significance decision would be reversed (i.e., reported as significant when computed is not, or vice versa).
get_decision_errors(x)get_decision_errors(x)
x |
An effectcheck object |
An effectcheck object containing only decision errors
results <- check_text("t(28) = 2.21, p = .035, d = 0.80") get_decision_errors(results)results <- check_text("t(28) = 2.21, p = .035, d = 0.80") get_decision_errors(results)
Returns information about an effect size family and its variants.
get_effect_family(effect_type)get_effect_family(effect_type)
effect_type |
The effect size type (e.g., "d", "eta2", "r") |
A list with family, variants, alternatives, and description
get_effect_family("d")get_effect_family("d")
Convenience function to extract only ERROR status results.
get_errors(x)get_errors(x)
x |
An effectcheck object |
An effectcheck object containing only errors
results <- check_text("t(28) = 2.21, p = .035, d = 0.80") get_errors(results)results <- check_text("t(28) = 2.21, p = .035, d = 0.80") get_errors(results)
Get same-type variants for a row
get_same_type_variants(x, row_index = 1)get_same_type_variants(x, row_index = 1)
x |
An effectcheck object |
row_index |
The row index |
A list of same-type variants with their values and metadata
res <- check_text("t(28) = 2.21, p = .035, d = 0.80") get_same_type_variants(res, 1)res <- check_text("t(28) = 2.21, p = .035, d = 0.80") get_same_type_variants(res, 1)
Returns metadata for a specific effect size variant type.
get_variant_metadata(variant_name)get_variant_metadata(variant_name)
variant_name |
The name of the variant (e.g., "d_ind", "dz", "eta2") |
A list with name, assumptions, when_to_use, and formula
get_variant_metadata("d_ind")get_variant_metadata("d_ind")
Extracts and parses the all_variants JSON structure for a given row.
get_variants(x, row_index = 1)get_variants(x, row_index = 1)
x |
An effectcheck object |
row_index |
The row index to extract variants from |
A list with same_type and alternatives sublists
res <- check_text("t(28) = 2.21, p = .035, d = 0.80") get_variants(res, 1)res <- check_text("t(28) = 2.21, p = .035, d = 0.80") get_variants(res, 1)
Convenience function to extract only WARN status results.
get_warnings(x)get_warnings(x)
x |
An effectcheck object |
An effectcheck object containing only warnings
results <- check_text("t(28) = 2.21, p = .035, d = 0.80") get_warnings(results)results <- check_text("t(28) = 2.21, p = .035, d = 0.80") get_warnings(results)
Test if object is an effectcheck object
is.effectcheck(x)is.effectcheck(x)
x |
Object to test |
Logical
res <- check_text("t(28) = 2.21, p = .035, d = 0.80") is.effectcheck(res)res <- check_text("t(28) = 2.21, p = .035, d = 0.80") is.effectcheck(res)
Extracts test statistics, effect sizes, confidence intervals, and sample sizes from APA-style text. Includes context window extraction for design inference.
parse_text(text, context_window_size = 2)parse_text(text, context_window_size = 2)
text |
Character vector of text to parse |
context_window_size |
Number of sentences before/after to capture (default 2) |
Tibble with parsed elements including context windows
parsed <- parse_text("t(28) = 2.21, p = .035, d = 0.80") parsed$test_type parsed$stat_valueparsed <- parse_text("t(28) = 2.21, p = .035, d = 0.80") parsed$test_type parsed$stat_value
Creates visualizations of effectcheck results.
## S3 method for class 'effectcheck' plot(x, type = c("status", "uncertainty", "test_type", "delta", "all"), ...)## S3 method for class 'effectcheck' plot(x, type = c("status", "uncertainty", "test_type", "delta", "all"), ...)
x |
An effectcheck object |
type |
Type of plot: "status", "uncertainty", "test_type", "delta", or "all" |
... |
Additional arguments passed to plotting functions |
Invisibly returns x.
res <- check_text("t(28) = 2.21, p = .035, d = 0.80") plot(res, type = "status")res <- check_text("t(28) = 2.21, p = .035, d = 0.80") plot(res, type = "status")
Displays a formatted summary of effectcheck results.
## S3 method for class 'effectcheck' print(x, short = TRUE, n = 10, ...)## S3 method for class 'effectcheck' print(x, short = TRUE, n = 10, ...)
x |
An effectcheck object |
short |
Logical, if TRUE show abbreviated output (default TRUE) |
n |
Maximum number of rows to display (default 10) |
... |
Additional arguments (ignored) |
Invisibly returns x.
res <- check_text("t(28) = 2.21, p = .035, d = 0.80") print(res)res <- check_text("t(28) = 2.21, p = .035, d = 0.80") print(res)
Print method for effectcheck comparison
## S3 method for class 'effectcheck_comparison' print(x, ...)## S3 method for class 'effectcheck_comparison' print(x, ...)
x |
An effectcheck_comparison object |
... |
Additional arguments (ignored) |
Invisibly returns x.
comp <- compare_with_statcheck("t(28) = 2.21, p = .035, d = 0.80") print(comp)comp <- compare_with_statcheck("t(28) = 2.21, p = .035, d = 0.80") print(comp)
Print method for summary.effectcheck objects
## S3 method for class 'summary.effectcheck' print(x, ...)## S3 method for class 'summary.effectcheck' print(x, ...)
x |
A summary.effectcheck object |
... |
Additional arguments (ignored) |
Invisibly returns x.
res <- check_text("t(28) = 2.21, p = .035, d = 0.80") s <- summary(res) print(s)res <- check_text("t(28) = 2.21, p = .035, d = 0.80") s <- summary(res) print(s)
Combine effectcheck objects
## S3 method for class 'effectcheck' rbind(...)## S3 method for class 'effectcheck' rbind(...)
... |
effectcheck objects to combine |
Combined effectcheck object
res1 <- check_text("t(28) = 2.21, p = .035") res2 <- check_text("F(1, 50) = 4.03, p = .049") combined <- rbind(res1, res2)res1 <- check_text("t(28) = 2.21, p = .035") res2 <- check_text("F(1, 50) = 4.03, p = .049") combined <- rbind(res1, res2)
Creates an HTML report with summary statistics, expandable sections, and uncertainty visualization.
render_report(res, out)render_report(res, out)
res |
tibble returned by check_text() / check_files() |
out |
output file path (html) |
Invisible path to the generated HTML report file.
res <- check_text("t(28) = 2.21, p = .035, d = 0.80") render_report(res, out = tempfile(fileext = ".html"))res <- check_text("t(28) = 2.21, p = .035, d = 0.80") render_report(res, out = tempfile(fileext = ".html"))
Provides comprehensive summary statistics for effectcheck results.
## S3 method for class 'effectcheck' summary(object, ...)## S3 method for class 'effectcheck' summary(object, ...)
object |
An effectcheck object |
... |
Additional arguments (ignored) |
A list of class "summary.effectcheck" containing summary statistics
res <- check_text("t(28) = 2.21, p = .035, d = 0.80") summary(res)res <- check_text("t(28) = 2.21, p = .035, d = 0.80") summary(res)