Why Replication Studies Are Hard, and What a Failed Replication Really Means

Replication means running a study again with new data to see whether the finding holds, and it is hard because methods are described incompletely, samples and settings differ, and published effects are often larger than reality. A single failed replication does not prove the original was false; it means two results disagree, and the actual scientific work is figuring out why.

Reproducibility versus replicability

These two words are often used interchangeably, but a careful reader keeps them apart. The National Academies drew a clean line. Reproducibility means taking the original data and the original analysis and getting the same numbers again. It is essentially a computational check, and when it fails, something is wrong with the code, the data handling, or the description of the analysis.

Replicability means running a new study, collecting fresh data, and asking whether the same finding appears again. This is a much higher bar, because a real effect should show up in new samples, but many things other than a false original can make a replication come out differently.

What a large replication project found

One influential effort repeated one hundred studies from three psychology journals, using high powered designs and the original materials where they could be obtained. The contrast was stark. Almost all of the original studies had reported statistically significant results, but only about a third of the replications did, and the replication effects were on average about half the size of the originals.

That result is often summarized as a crisis, but the more useful reading is quantitative. It does not say the original findings were all wrong. It says that published effects tend to look larger and more certain than they are, and that a field which never checks will not know which of its results are solid.

Why replication is genuinely hard

Part of the difficulty is practical. Methods sections are compressed, and the details that decide whether an experiment works, the exact materials, the timing, the population, the unspoken laboratory know-how, are frequently missing. A replication team can follow the printed recipe faithfully and still not be running the same study.

Part is contextual. Effects can be real but conditional, holding in one population or setting and not another. When a replication in a different context comes out flat, that can be genuine information about the limits of the finding rather than evidence that the original was an error. Separating a fragile result from a context-bound one takes more than a single repeat.

Why effects tend to shrink

Two statistical facts explain much of the shrinkage. The first is regression to the mean. An initial study that happened to catch a high estimate by chance will, on average, produce a smaller estimate the next time, simply because extreme values are not typical.

The second is the winner's curse of selective publication. When mainly striking results get published, the published estimate is drawn from the upper tail of what the data could have shown, so it overstates the true effect. A replication, run regardless of how it turns out, lands closer to reality and therefore looks disappointing by comparison. Neither mechanism requires anyone to have done anything wrong.

What a failed replication actually tells you

A single divergent result is a disagreement, not a refutation. The honest response is to ask why the two studies differ: was the original a false positive, was the replication underpowered, did the context change, or was some crucial method never captured on the page. Each of those has a different remedy.

This is why the strongest evidence comes from many replications rather than one, and from teams that register their plan in advance so the analysis cannot drift toward a convenient answer. Replication is a process of triangulation, and one data point, in either direction, rarely settles the question.

Why replication is worth the trouble

Replication is slow, unglamorous, and often unrewarded, which is exactly why it has been neglected. Its value is that it converts a published claim into something the field actually knows, by testing whether the result survives contact with new data and new hands.

The goal is not to catch people out. It is to build a literature where the surviving findings are the ones you can lean on. A discipline that replicates its important results is not admitting weakness; it is doing the part of science that turns interesting into reliable.