When Real-World Evidence Helps and When It Misleads

Real-world evidence is trustworthy when the question was specified before anyone looked at the data, when the groups being compared would have been clinically interchangeable at the moment of the decision, and when the measured outcome is hard to fake. It misleads in the opposite case: a question shaped by the answer, groups that differ in ways nobody recorded, and an outcome that quietly tracks who was sick to begin with. The hard part is not the statistics. It is knowing which of those two situations you are in.

By real-world evidence I mean clinical conclusions drawn from data generated during ordinary care rather than inside a randomized experiment: electronic health records, insurance claims, device logs, registries, pharmacy refills, wearables. Randomization is the one tool that breaks the link between why a patient got a treatment and how that patient was going to do anyway. Remove it and that link comes back, often invisibly.

What can real-world evidence actually answer well?

It answers questions about the messy middle of practice, where trials rarely go. Trials enroll selected people, watch them closely, and stop after a year or two. Real care includes the eighty-year-old on six other drugs and the patient who skips half her doses. Observational data sees all of them, which is why it helps with spotting rare adverse events and asking whether a result proven in a pristine trial survives contact with ordinary clinics.

There is also a class of questions where a randomized trial would be unethical or impossible, and here observational data is the only honest option. You cannot randomize people to a hurricane, a workplace exposure, or decades of a dietary pattern. Some of the most secure knowledge in medicine, the link between smoking and lung cancer, was built this way, because the effect was enormous and every plausible alternative explanation was ruled out.

Why does observational data mislead, and what is confounding?

Confounding happens when something that influences who gets a treatment also influences the outcome, so the treatment takes credit, or blame, for an effect it did not cause. A well-known example shaped thinking for years. Women who took a certain hormone therapy appeared to have less heart disease in observational studies, so it looked protective. When the question was finally put to a randomized trial, the protection was not there, and for some groups the direction reversed. The data had measured something real, just not what most people assumed. The women who took the therapy tended to be wealthier and more health-engaged, so the drug had been keeping company with good health, not producing it.

This pattern is common enough to have a name. The healthy user effect describes how people who stick with any preventive treatment also tend to exercise, eat better, and show up for screening, so the treatment looks good because its takers were already on a better path. Confounding by indication runs the other way: the sickest patients get the strongest drugs, so a strong drug can look dangerous when it was the underlying illness doing the harm.

Statistics can adjust for confounders, but only the ones you measured. The hardest version is the variable nobody wrote down: frailty, motivation, the things a clinician senses in a room and an algorithm never sees. You cannot correct for a column that is not in your table.

How do selection effects quietly distort the picture?

Selection bias is about who ends up in your data at all, and who falls out before the outcome is counted. It is subtler than confounding because it can survive even perfect measurement of everyone who remains. Study patients still in the registry two years after starting a drug, and you have already excluded everyone who quit because the drug failed or harmed them. The survivors make it look good while the people who would tell the other half of the story are gone.

Immortal time bias is the version that catches careful people. Suppose you define a group by the fact that they filled a second prescription. To fill it, a patient had to survive long enough to do so, so you have built survival into the definition of the group and then marveled that the group survives. Whenever group membership depends on something that happened after the starting line, suspect you have guaranteed part of the result.

Why does a pre-specified question matter so much?

A clear question, written down before the analysis, is the best protection against fooling yourself with real-world data. Health datasets are wide, and with enough variables, subgroups, and outcome definitions, you can almost always find a comparison that reaches statistical significance by chance alone. If you let the data suggest the question after you have seen the answer, you are no longer testing a hypothesis. You are decorating a coincidence.

Pre-specification fixes this by committing you in advance: this is the population, the comparison, the outcome, the analysis. A finding that holds up under those rules earns far more trust than one excavated by trying everything until something glows. A practical version is to imagine the randomized trial you wish you could run, then build your analysis to imitate it as closely as the data allows.

How should you read a real-world finding?

Trust the direction and plausibility of an effect more than its precise size, and a large, biologically sensible effect more than a small statistical one. A small relative difference is exactly the size that unmeasured confounding can manufacture on its own. A very large effect, consistent across populations, is harder for hidden bias to explain.

In our own work building decision-support tools, I leaned on this constantly. Real-world usage data was valuable for understanding how clinicians behaved and where a workflow broke down. But for the central claim, whether the system improved patient outcomes versus standard of care, we ran a multi-clinic randomized controlled trial, EASY-1 (NCT03258268), because that was the only design that could carry it. Real-world evidence and randomized trials are not rivals. They answer different questions, and the craft is matching design to claim.

This article is educational and not medical advice. If you are weighing a treatment decision, talk it through with your own clinician.