How Many Patients It Takes to Build a Prediction Model

The familiar rule of ten outcome events per candidate variable is neither necessary nor sufficient for building a reliable prediction model. Modern sample-size methods instead choose a number of participants and events that keeps overfitting small, keeps the model's optimism low, and estimates the overall risk precisely. Depending on the setting, the required number of events per variable can be well below ten or several times higher.

Where the ten-per-variable rule came from

For years, the working rule for building a risk model was ten outcome events for every candidate predictor. It was easy to remember and grew out of simulation work on regression stability. The trouble is that it was always a rough approximation, and treating it as a hard law leads people to build models on samples that are far too small, and occasionally larger than they need to be.

Why too small a sample bites later

When a model is fit on too few events, it latches onto noise. Coefficients get exaggerated, and the model looks excellent on the data that trained it. Then it meets new patients and its performance collapses, a gap called optimism. To counter this, statisticians apply shrinkage, pulling coefficients toward zero so the model generalizes better. A model that needs heavy shrinkage was overfit, and in effect it shrinks toward a simpler, less confident version of itself.

The modern criteria

A more careful approach chooses the sample so that three things hold at once. First, the expected shrinkage is small, with a global shrinkage factor of at least about nine-tenths, meaning little overfitting. Second, the difference between the model's apparent performance and its optimism-adjusted performance is small. Third, the overall risk in the population is estimated precisely. Meeting all three, rather than counting events per variable, defines the minimum sample.

It cuts both ways

Applying these criteria to real problems shows why the old rule was unreliable. A published diagnostic model for one infectious disease needed only about five events per predictor, comfortably below ten. A prognostic model for recurrent blood clots needed more than twenty. The required number depends on how many predictors you consider and on how much signal the model is expected to carry, which you estimate from earlier studies.

What this means for a reader

You do not need to run the calculation to use the idea. When a paper introduces a new risk score, look at how many events it was built on relative to the number of predictors considered, including the ones that were tested and dropped. Ask whether the authors justified their sample size with a real calculation and reported the shrinkage or optimism. A model developed on a thin sample, with no shrinkage and no external test, deserves caution no matter how good its headline numbers look.