What a Meta-Analysis Cannot Fix

A meta-analysis cannot fix three things, and pretending otherwise is how careful readers get fooled by careful arithmetic. It cannot turn biased or low-quality input studies into trustworthy ones. It cannot reconcile real disagreement between studies into a single honest number. And it cannot recover the results that were never published in the first place. Pooling combines evidence; it does not launder it. The math can be flawless while the conclusion is wrong, because the math inherits whatever flaws arrived with the studies. This article explains a method for general education; it is not medical advice, and for decisions about your own care you should talk with a qualified clinician who knows your history.

The promise of a meta-analysis is seductive. Many studies feel sturdier than one, and a pooled estimate with a narrow interval looks like the last word. That impression is worth questioning before you trust it.

Why pooling does not improve a study

The first myth worth retiring is that combining studies makes each one better. It does not. A meta-analysis multiplies sample size, which buys precision, but precision is not accuracy. If every input study tilts the same direction for the same reason, pooling them produces a tighter estimate of the wrong answer.

Think of bias as a thumb on the scale that always presses one way. Random error scatters in every direction, so averaging many studies cancels it out, and that is the genuine gift of synthesis. Systematic error does not scatter. When ten trials share an unblinded outcome assessment, or a comparison group that was never truly comparable, the tenth study does not correct the first. It repeats it. The pooled estimate then lands confidently on a result that was bent from the start.

This is why a serious review weights studies for risk of bias rather than counting them as equal votes. A vote-counting synthesis treats a small, poorly controlled study as the equal of a large, rigorous one, and the cleaner picture it produces is an artifact of the counting, not of the evidence. Even careful weighting has a ceiling. If the whole pool is built from one weak design, weighting cannot manufacture a strong study that nobody ran.

True heterogeneity is a finding, not a flaw to average away

The second thing pooling cannot fix is genuine disagreement among the studies. Heterogeneity is the word for that disagreement, and it comes in two kinds that often get blurred together. Some of it is statistical noise, the ordinary scatter you expect when you measure the same thing many times. Some of it is real, meaning the studies disagree because they were measuring meaningfully different things.

When the disagreement is real, a single pooled number describes none of the studies honestly. Picture two trials that point in opposite directions because they enrolled different populations or assessed the outcome at different times. Averaging them yields a figure in the middle that fits neither, like reporting the average temperature of a person with one hand in ice water and the other in a hot bath. The number is arithmetically true and clinically empty.

The right response to real heterogeneity is to explain it, usually by examining pre-specified subgroups, rather than to pick a statistical model that quietly assumes the studies agree more than they do. Differences in how an outcome was measured, or in who was enrolled, are not a nuisance to smooth over. They are often the most interesting part of the question. A review that finds high heterogeneity and prints one confident result anyway has buried its most important finding underneath its headline.

Publication bias removes evidence before counting begins

The third and quietest failure is the one the arithmetic literally cannot see. Publication bias is the tendency for studies with positive, striking results to reach print while null or disappointing ones stay in a file drawer. A meta-analysis can only pool the studies it can find, so when the unflattering results were never published, the synthesis is averaging a sample that was filtered before anyone touched a calculator.

No statistical method recovers data that does not exist. Funnel plots and related tests can raise a suspicion that small null studies are missing, and a good review runs them, but a suspicion is not the missing studies themselves. The most rigorous synthesis imaginable, applied to a literature where half the relevant trials were never reported, will produce a confident estimate of a biased subset. The confidence is real. The estimate is not the truth.

This is the deepest sense in which garbage in produces garbage out, because here the garbage is invisible. A reader sees twenty published studies pointing the same way and feels reassured by the agreement. That agreement may be the signal, or it may be the shape of what survived the path to publication. Pre-registration of trials exists to make the unpublished visible, and its slow spread is one of the more hopeful changes in research practice.

So what is a meta-analysis good for

None of this means evidence synthesis is a parlor trick. Done well, on a complete and fairly searched literature of sound studies, a meta-analysis is among the strongest tools available, because accumulation can reveal a real effect that no single underpowered study could detect. The method earns its high standing when the inputs deserve it.

The discipline is to ask what the pool is made of before trusting what comes out. Were the studies at low risk of bias? Did they genuinely agree? Is there reason to believe the whole literature was found, rather than only the flattering part of it? When the answer to all three is yes, the result means what it appears to mean. When the answer to any is no, the same result is an average wearing a lab coat, and a narrow confidence interval makes a filtered finding look settled.

A meta-analysis is a mirror, not a filter. It reflects the literature it is given with more precision than any single study, and precision applied to a distorted reflection produces a sharper distortion. Respecting the method means respecting its limits, and reading the methods before the conclusion is how you tell which kind of reflection you are looking at.