Testing Many Outcomes at Once: How Trials Keep False Positives in Check

Each additional statistical test in a trial, whether an extra endpoint, a subgroup, or an interim peek at the data, adds another chance for a random finding to cross the significance line. To keep the overall false positive rate near five percent, trials divide their alpha budget in advance, test endpoints in a fixed order, or use an alpha spending function for interim analyses. When you read a result, the useful question is how many tests were run and whether the analysis plan accounted for them.

The core problem: more tests, more false alarms

Statistical significance is usually set so that a single test has about a five percent chance of a false positive when nothing real is going on. That is tolerable for one test. The trouble is that trials rarely run just one. Add a second endpoint, a handful of subgroups, and a couple of interim looks, and you are running many tests, each with its own five percent chance of crying wolf.

The arithmetic is unforgiving. With independent tests, the probability that at least one crosses the line by chance is one minus the chance they all stay quiet, and that climbs fast. Twenty independent tests at the usual threshold give better than even odds of at least one false positive. So a lone impressive p value, pulled from a trial that quietly ran many comparisons, tells you much less than it appears to.

Familywise error rate and the alpha budget

The quantity that matters once you are running several tests is the familywise error rate: the probability of making at least one false positive across the whole family of tests. Confirmatory trials are expected to hold this rate near the target, usually five percent, no matter how many endpoints are in play.

The simplest way to picture the fix is a budget. The trial starts with a fixed amount of alpha, and every test it wants to count as confirmatory has to be paid for out of that budget. Split it evenly across endpoints, weight it toward the ones that matter most, or spend it in sequence; the rule is that the total cannot exceed what you started with. An endpoint that was never allocated any alpha is simply not a confirmatory finding, whatever its p value happens to be.

Hierarchical and fixed-sequence testing

One efficient way to spend the budget is to test in a prespecified order. In a fixed sequence, the trial ranks its hypotheses in advance, tests the first at the full threshold, and is allowed to proceed to the second only if the first succeeds. As long as the chain holds, each test can use the full alpha, because the ordering itself controls the error rate.

The catch is that the chain breaks at the first failure. If the primary endpoint does not reach significance, the secondary endpoints below it cannot be declared confirmatory, even when their own numbers look good. This is why the prespecified testing order deserves as much attention as the p values themselves: a strong looking secondary result sitting below a failed primary is not what it seems.

Interim looks and alpha spending functions

Trials often peek at the data before the end, to stop early for clear benefit, for harm, or for futility. Each peek is another test, so naive repeated looks inflate the false positive rate the same way multiple endpoints do. Looking twice at the usual threshold pushes the real error rate up toward ten percent rather than five.

The solution is an alpha spending function, which decides in advance how much of the alpha budget each interim look may use. Conservative schemes, such as the O'Brien and Fleming style, spend almost nothing early, so stopping at an interim look requires a striking effect, and they leave nearly the full threshold for the final analysis. The flexible version developed by Lan and DeMets lets a trial add or move looks without fixing their exact timing ahead of time, as long as the total spend is capped. When a trial stopped early, it is worth confirming it used such a boundary rather than simply halting when the number looked good.

Where multiplicity hides

Multiplicity is easiest to spot in a tidy list of endpoints and easiest to miss everywhere else. Secondary and exploratory endpoints are common hiding places: a paper may control alpha carefully for its primary outcome and then report a dozen secondary comparisons at the usual threshold, any of which can reach significance by chance.

Subgroups are the other classic. Slice a trial by age, sex, disease severity, and a few biomarkers, and some subgroup will look like it responded, purely from noise. Such findings are not worthless, but they are hypothesis generating, not confirmatory, and a result that was not part of the prespecified, alpha controlled plan should be read as a lead to test next, not a settled conclusion.

Reading a trial with multiplicity in mind

You can do a lot with a few questions. How many endpoints and comparisons did the trial actually run, and was there a stated plan for controlling the overall error rate? Was the impressive result the prespecified primary outcome, or a secondary or subgroup finding further down the list?

If the trial peeked early, did it use an alpha spending boundary? And when a subgroup or secondary endpoint becomes the headline, is it framed as a hypothesis to confirm or sold as a finding? A trial that names its primary endpoint, controls its alpha, and keeps secondary results in their place is far easier to trust than one that presents its single best p value and hopes you will not count the tests behind it.