What a P-Value Really Means, and the Four Things People Think It Means but It Does Not

A p-value answers one narrow question: if nothing real were going on, how surprising would data this extreme be? Formally, it is the probability of observing a result at least as extreme as the one you got, assuming the null hypothesis is true. That is the whole definition. It is not the probability that your finding is true, not the probability that the null is false, not a measure of how large or how important the effect is, and a number below 0.05 is not a stamp that says the result is real. This is a methods article, not medical advice; for anything about your own care, talk with a clinician who knows your history.

I have spent years on both ends of this number. As a reviewer I have read manuscripts where a p of 0.04 was carrying the whole argument, and as an author, including a Diabetes Care meta-analysis I co-wrote, I have had to decide what a borderline value was allowed to claim. The number is rarely the problem. What people load onto it is.

What is a p-value, in one plain sentence

Here is the quotable version. A p-value is the chance of getting data at least as extreme as yours if the effect you are testing for does not exist. A small p-value means your data would be unusual in a world with no effect, which makes that no-effect world a little harder to believe. That is all it does. It nudges you to doubt the null. It never confirms the alternative.

The mechanism is worth picturing, because the picture prevents most of the errors. You assume the boring story is true: the drug does nothing, the groups are the same, the gene is unlinked. Then you ask how often pure chance, under that story, would hand you a result this dramatic. If the honest answer is "rarely," you have grounds to suspect the boring story is wrong. You do not have a probability attached to your exciting story. You have a measure of how poorly the data sit with the boring one.

Why a p-value is not the probability the result is true

This is the misreading I see most, and it is seductive because the plain-English sentence feels symmetric when it is not. People read "p equals 0.03" as "there is a 3 percent chance this is a fluke and a 97 percent chance it is real." That flips the conditional. The p-value gives the probability of the data given the null, not the probability of the null given the data, and those two are not interchangeable.

A courtroom analogy keeps me honest. A trial assumes innocence, then asks how unlikely the evidence would be if the defendant were innocent. A pattern very unlikely under innocence raises suspicion, but it does not by itself tell you the probability of guilt, because that depends on what you knew before the evidence arrived. To get from "the data are surprising under the null" to "the finding is probably true," you need the prior plausibility of the claim, and the p-value contains none of it. So if you test a far-fetched hypothesis, most of your "significant" results will be false alarms, because a low prior swamps a modest p-value. The number behaved as designed. The expectation placed on it did not.

Why a small p-value does not mean a large or important effect

A p-value measures how cleanly you can rule out chance, not how big the thing you found is. Those drift apart the moment your sample grows. With tens of thousands of participants, a difference too small to matter to any one person can post a stunningly small p-value, because large samples make even trivial effects detectable. Significance and importance are different axes, and conflating them dresses a true-but-tiny effect up as a discovery.

The reverse trap is just as common and, in my view, more costly. A real and clinically meaningful effect can land at p equals 0.08 because the study was too small to pin it down, and a reflexive reader files it under "no effect." A large p-value says the data are compatible with the null, not that the null is true. Absence of significance is not evidence of absence. This is why I look past the p to the effect size and its confidence interval, which carries the magnitude the p-value throws away. A result can be significant and trivial, or non-significant and genuinely promising but underpowered, and only the size and its uncertainty separate the two.

Why 0.05 is a convention, not a law of nature

The 0.05 threshold is a historical convenience that hardened into a rule nobody quite voted for. There is nothing magic about it. A p of 0.049 and a p of 0.051 describe almost identical evidence, yet a strict cutoff treats one as a triumph and the other as a failure. The threshold was meant as a rough screen, and reading it as a verdict invites two bad habits.

The first is dichotomy: collapsing a continuous measure of surprise into "significant" or "not," then reasoning as if those boxes were real categories of nature. The second is the file drawer, where results that clear the line get written up and results that miss it quietly disappear, so the published record drifts toward whatever passed an arbitrary gate.

The quiet damage of p-hacking

There is a subtler failure that no single p-value can reveal, and it is rarely born of bad faith. If you test many outcomes, slice the data into many subgroups, or keep collecting until the number dips under 0.05, you will eventually find a "significant" result even when nothing is there. The reported p-value assumes one clean prespecified test, so it cannot account for the quiet attempts behind it. The integrity of the number depends on the process that produced it.

How I read a p-value as a reviewer

When I review, the p-value is one input among several, and rarely the deciding one. I want to know what was prespecified, how many comparisons were run, how big the effect is, and how wide its confidence interval is. A small p in a registered, single-outcome trial earns my attention. The same number at the end of a fishing expedition earns my suspicion.

So treat the p-value as a smoke detector, not a fire inspector. It flags that something might warrant a closer look, and it cannot rule on truth, size, or importance. Ask what world the number assumes, what it can say once you grant that world, and what the surrounding evidence makes plausible. The field is genuinely hard, and the people producing these numbers are mostly working in good faith under real pressure to publish. The fix is not to fear the p-value, but to stop asking it for answers it was never built to give.