What Peer Review Can and Cannot Catch

Peer review reliably catches the things a careful reader can verify from the manuscript: muddled writing, claims the data do not support, missing analyses, ignored prior work, and conclusions that overreach the design. It rarely catches the things that require access reviewers do not have: outright fabrication, selective reporting of experiments that were quietly dropped, and whether the result holds up when someone else repeats it. So passing peer review means a paper is competent and defensible on its own terms. It does not mean the paper is true. Holding both facts at once is the most useful thing a reader can do with the word "peer-reviewed." This is an article about how science is vetted, not medical advice; for decisions about your own care, talk with a clinician who knows your history.

I have sat in most of the seats around this process. I have submitted papers and absorbed the critiques, reviewed manuscripts and grants (including for the Pivotal Philanthropies Action for Women's Health initiative), and watched my own work get sharpened by reviewers who saw what I had stopped seeing. The system is run almost entirely by volunteers reading on evenings and weekends, and it works better than its reputation suggests. It just works at a specific thing, and trouble starts when we ask it to certify what it was never built to certify.

What does peer review actually do?

Peer review is a structured critique of a manuscript by independent experts before publication, meant to check whether the methods support the claims and whether the work is sound enough to enter the record.

Peer review tests whether a paper is internally honest and defensible. It does not, and cannot, test whether the underlying experiments happened the way the authors say they did.

The distinction sounds pedantic until you see where errors come from. A reviewer reads what is on the page. If that page is coherent and consistent with its figures, the reviewer has little leverage to discover that a figure was assembled rather than measured. The process audits the argument, not the laboratory.

What peer review catches well

The honest account begins with credit, because the system earns its reputation at several things.

Reviewers are good at catching a conclusion that outruns its design. When a paper built on an observational cohort starts using causal verbs, an alert reviewer flags it, and the authors either soften the language or add a caveat. That single correction, repeated across thousands of manuscripts a year, quietly raises the quality of the literature.

They catch missing comparisons. A treatment that looks impressive against nothing often looks ordinary against a fair current alternative, and a good reviewer asks for the harder comparison. They catch statistics applied to the wrong situation, or a subgroup result presented as if it were the planned analysis. They catch the quieter sin of ignoring prior work, when a paper claims novelty that the field settled a decade ago and a reviewer who knows that history says so first.

And reviewers improve clarity in ways readers never see. A muddled methods section gets rewritten until a stranger could follow it, and an overstated abstract gets pulled back to match the body. Most of what review produces is invisible by design, because it happens in the gap between the version submitted and the version published.

What slips through, and why

The failures are not failures of effort. They are failures of access, and naming them precisely is how you read the literature wisely.

Fabrication and the data nobody sees

A reviewer evaluates the data presented, not the data collected. If numbers are invented or images manipulated with any skill, the manuscript can read as flawless, because internal consistency is exactly what a fabricator optimizes for. The honest researcher and the dishonest one submit documents that look the same from the outside. Detecting fraud usually requires raw files, lab notebooks, or a later attempt to reproduce the work, none of which a reviewer holds. The process was never designed as a fraud detector, and treating it as one sets everyone up for disappointment.

Selective reporting and the file drawer

A subtler problem is the experiment that was run, did not work, and quietly disappeared before submission. A reviewer sees the analyses the authors chose to show, not the ones left on the cutting room floor. If a team measured ten outcomes and reported the two that reached significance, the paper can be entirely truthful about those two and still paint a misleading picture. Pre-registration helps, because it lets a reviewer compare what was promised against what was delivered, but it is far from universal.

Whether the result will replicate

Replication is whether an independent team, running the same experiment, gets a compatible answer. It is the real test of a scientific claim, and it happens after publication, outside the reach of any reviewer. A result can be cleanly reviewed and still fail to replicate, because the original finding was a fluke of one sample, one batch of reagents, or one population. Peer review checks whether a claim is plausible given the evidence shown. Only replication checks whether it is durable.

The narrow window of expertise

Two or three reviewers cannot span every method in a modern paper. A study might combine a clinical trial, a genetics analysis, and a machine-learning model, and the reviewer expert in one of those may be guessing politely about the rest. I have seen my own field, the genetics of complex disease, evaluated by careful people who knew the biology but not the statistical machinery underneath, and the reverse. No one is at fault. The work has outgrown the idea that a few generalists can vouch for all of it.

How to read "peer-reviewed" honestly

Treat publication as a floor, not a ceiling. A peer-reviewed paper has cleared a bar of competence and survived scrutiny from people with no stake in flattering it, which is worth something real. It is the beginning of the paper's life in the literature, not the end of the argument about whether it is right.

So I read a single study as a serious proposal rather than a settled fact, and I look for whether anyone has since tried to repeat the work. A claim that has replicated across independent groups has passed a test no amount of pre-publication review can substitute for. A striking result that no one has reproduced is interesting and unproven at the same time, and both halves of that sentence matter.

A fair word for the people who do it

It would be easy to read all this as an indictment, and that would be the wrong lesson. The remarkable thing is how well an unpaid, decentralized system of busy experts keeps the literature as sound as it is. Reviewers catch real errors every day that would otherwise mislead patients and colleagues, for the oldest reason in science, which is that they would want the same care taken with their own work. The fix for these limits is not cynicism. It is layering review with pre-registration, data sharing, and replication, so each method covers what the others cannot. Understanding what peer review can and cannot catch is what lets you respect it and read past it at the same time.