Checking a Paper's Own Arithmetic: What statcheck and GRIM Reveal

Some integrity checks need no raw data at all; they test whether a paper's own numbers are internally consistent. The statcheck tool recomputes p values from the reported test statistic and degrees of freedom, and the GRIM test checks whether a reported mean is even possible given the sample size. Both flag impossibilities that deserve a second look, though a flag signals an inconsistency, not proof of wrongdoing.

Consistency checks need no raw data

Most people assume that catching an error in a paper requires the original dataset. For a whole class of problems, it does not. A paper's reported numbers have to be consistent with each other, and that internal consistency can be checked from the text alone.

This is a quietly powerful idea. If a stated average is impossible for the given sample size, or a p value does not match the statistic it came from, something is wrong regardless of what the raw data would show. Two simple tools built entire methods on that observation.

How statcheck works

When researchers report a statistical test, they usually give the test statistic, the degrees of freedom, and a p value. Those three are linked by a fixed formula, so any two of them determine the third. The statcheck tool exploits this by recomputing the p value and comparing it to the one printed.

A screen of more than a quarter of a million reported p values across major psychology journals found that about half of the papers using significance testing contained at least one result whose p value was inconsistent with its own statistic. Roughly one in eight papers contained a grossly inconsistent value, meaning the mismatch could have changed whether a result was called significant.

How the GRIM test works

The GRIM test is even simpler, and it applies when a study reports the mean of items answered in whole numbers, such as a scale from one to seven. With a given number of participants, only certain averages are mathematically possible, because the total has to be a whole number divided by the sample size.

If a paper reports a mean that no combination of whole-number responses could produce for the stated sample size, that mean cannot be right as printed. Applying this check to a set of published papers surfaced a surprising number of impossible values that had passed review unnoticed.

What a flag means and does not mean

These flags call for restraint. An inconsistency is a discrepancy, not a diagnosis. Many flagged values trace to innocent causes: a transposed digit, a rounding convention, a copy error between the analysis and the manuscript, or a detail the tool could not parse.

So the correct response to a flag is a closer look, not an accusation. The tools narrow attention to the numbers worth checking. They cannot tell you why a number is off, and the difference between a typo and a fabricated result can only be settled by examining the underlying work.

Why significant results were flagged more often

One pattern in the statcheck findings deserves attention. Gross inconsistencies were more common among p values reported as significant than among those reported as nonsignificant.

That asymmetry is what you would expect if there were a subtle pull toward significance, whether through wishful rounding, error, or motivated reading of a borderline result. On its own it proves nothing about any single paper, but across a literature it is a fingerprint worth noticing, and it is exactly the kind of signal these consistency checks are good at surfacing.

How these fit a culture of openness

Consistency tools work best as part of a wider set of habits rather than as a hunt for villains. The same researchers who mapped the reporting errors suggested using such checks during writing and peer review, so problems are caught before publication rather than after.

They also pointed to broader remedies: sharing data so results can be verified directly, and having a co-author independently rerun the analysis before submission. Seen this way, statcheck and the GRIM test are not weapons but proofreading for the numbers, part of the ordinary care that keeps a literature trustworthy.