Multiple Testing: Why Twenty Questions of One Dataset Produce a False Positive

Ask one dataset enough questions and one of them will answer yes by luck alone, even when nothing real is there. That is the multiple-comparisons problem in a sentence. A single statistical test set at the usual threshold accepts a one-in-twenty chance of a false alarm, so if you run twenty independent tests on the same data, you should expect roughly one to look significant purely by chance. The fix is not exotic: decide in advance which question matters most, count how many questions you asked, and adjust the bar accordingly. This piece is educational and not medical advice; for decisions about your own care, talk with your own clinician.

I have felt the pull of this problem from the inside. In our Diabetes Care systematic review and meta-analysis of ethnic differences in insulin sensitivity and insulin response, and across my doctoral work on the genetics of type 2 diabetes, the temptation is always the same: the data are rich, the subgroups are many, and something will glitter if you tilt it toward the light.

Why does asking more questions guarantee a false positive?

Start with what a p-value threshold promises. When a study calls a result significant, it accepts a fixed rate of being wrong when there is truly nothing to find. Set that rate at one in twenty, and a single test is a fair coin weighted heavily toward honesty. The trouble is that the promise is per test, not per study.

A single test carries a small chance of a false positive. Add a second, and the odds that at least one cries wolf edge upward. By the twentieth, a false alarm somewhere in the batch stops being an unlucky accident and becomes the expected outcome. Hold onto the image: each new question is another ticket in a raffle whose prize is a spurious finding. Buy enough tickets and you will win, and winning here means being fooled.

Nothing about this requires bad faith. A careful team measuring fifteen outcomes across four subgroups at three time points has quietly created hundreds of comparisons without a single dishonest act. The false positives arrive on their own, invited by the sheer number of chances.

What is the difference between family-wise error and false discovery rate?

Once you accept that many tests breed false alarms, the question becomes what you want to control. There are two honest answers.

Controlling the family-wise error rate

The strict approach asks about the chance of making even one false positive across the entire family of tests. Controlling the family-wise error rate holds that chance down to the usual level for the whole batch, not for each test alone. The most familiar way to do this raises the bar for every test in proportion to how many you ran, so twenty tests each face a threshold roughly twenty times stricter. When a single false claim would be costly, that conservatism is a feature: you would rather miss a real effect than announce a fake one.

Controlling the false discovery rate

The other approach is gentler and better suited to discovery. The false discovery rate does not try to prevent every false positive. It controls the expected proportion of false positives among the results you flag as significant. If a genomics screen flags a hundred variants and the false discovery rate is set at one in twenty, you accept that around five are probably noise, a reasonable price when the goal is to generate leads for the next experiment. Choosing between the two is a statement about what kind of mistake you can live with.

Why do prespecified primary endpoints matter so much?

The cleanest defense against multiple testing is not a correction applied afterward. It is a decision made before any data arrive. A trial that names one primary endpoint in advance, and registers it publicly, has committed to the single question it will be judged on. Everything else becomes secondary or exploratory, labeled as such.

The danger runs deeper than the count of tests. It lives in the freedom to choose which one to celebrate after seeing the results. If a study can measure ten things and then present whichever came out best as though it were the plan all along, no correction can rescue it, because the selection itself is the bias. Prespecification fixes the target before the arrow is loosed, so a hit means something.

I have worked inside this discipline. With EASY Diabetes, an AI clinical decision-support system for type 2 diabetes that I co-developed, we ran EASY-1, a registered randomized controlled trial (NCT03258268) that compared the system against standard of care. Naming the primary endpoint ahead of time was not paperwork. It was the difference between testing a hypothesis and dressing up a dataset.

How do you spot uncorrected subgroup fishing?

You do not need the mathematics to read defensively. Subgroup fishing has a recognizable shape.

The first tell is a headline built on a slice rather than the whole. A trial misses its main target but reports that the treatment worked wonderfully in one age band, or one sex, or one country. A real effect in a subgroup is a hypothesis for the next study, not a conclusion from this one, especially when the overall result was flat.

Watch, too, for an unstated denominator. If a paper reports three exciting positive findings, the honest question is how many comparisons were run in total. Three wins out of five tests is interesting. Three out of ninety is what chance alone would hand you. When the total count is missing, the significant results cannot be interpreted, and that silence is itself informative.

A third pattern is significance that appears without any mention of correction. When a study reports many outcomes, each at the naive threshold, with no adjustment and no prespecified primary, treat the whole thing as hypothesis-generating rather than conclusive. That is no insult to the researchers. It is the correct weight for evidence produced that way.

The reader's job is not to distrust every positive finding, since many are real and hard-won. It is to ask a plain question of any striking result: how many other questions were asked of the same data, and did anyone account for them. A study that reports its primary endpoint, its total comparisons, and its correction method is being honest about the raffle it entered. One that shows you only its winning ticket wants you to forget the raffle happened.