How Regulators Grade Clinical Evidence, and Why the Bar Moves With Risk

When a regulator or assessor reads a clinical dossier, they grade it on three axes at once: how well each study was done, how closely it matches the people and the decision in front of them, and whether the whole body of evidence points one way. They then set the height of the bar by the risk of the product and what it claims to do, so a low-stakes wellness feature and an implant that sustains life are not asked to clear the same fence. The grade is not a tally of studies. It is a judgment about whether the evidence, taken together, supports the specific use being claimed. This is a methods and policy article, not medical advice; for your own care, talk with a clinician who knows your history.

I have spent time on both sides of this exchange. I completed FDA Clinical Investigator training, hold training in medical device regulations from KTH, and worked in global drug development as an international medical manager, where the daily task was stating exactly what a result licensed us to claim. One lesson stuck. Assessors are not impressed by volume. They are persuaded by fit.

Study quality comes first, but it is not the whole grade

Quality is the question of whether a study earned the result it reports. An assessor looks at how participants were assigned, whether the people measuring outcomes knew who got what, how many dropped out, and whether the analysis counted everyone who started. A randomized, blinded design that follows its pre-registered plan resists the biases that make weaker designs flatter the treatment.

A common error is to treat the design label as the grade. A randomized trial run carelessly can be less trustworthy than a well-built observational study, and assessors read the conduct rather than the badge. What matters is whether the methods controlled the specific ways this kind of study tends to mislead.

Relevance asks whether the study answers the right question

Relevance is the fit between the evidence and the decision being made. A flawless trial can still be the wrong evidence if it studied a different population, a different version of the product, a different dose, or an outcome that does not match the claim. Assessors call this directness, and it often does more work in the final judgment than quality does.

The gap usually hides in the details. A study may have enrolled younger and healthier volunteers than the people who will actually use the product, or measured a short-term laboratory marker when the claim is about something a person can feel over a year. A meta-analysis I co-authored in Diabetes Care examined how the relationship between insulin sensitivity and insulin response differs across ethnic groups, and its practical message was that a result established in one population cannot be assumed to transfer cleanly to another. Relevance is where that caution becomes a regulatory question.

Totality is the test a single trial cannot pass

Totality is the demand that the whole body of evidence point in a consistent direction. One striking trial is a claim awaiting confirmation. Several studies of different designs, in different settings, converging on a similar answer are far harder to explain away as chance or the quirk of one site. Assessors weigh consistency, the size of the effect, the dose-response pattern, and biological plausibility together rather than reading any single number alone.

Totality also means counting the evidence that did not flatter the product. Assessors look for the studies that were started and never reported, the outcomes measured and quietly dropped, and the subgroups where the effect disappeared. A dossier that presents only its best results has told the assessor less than it thinks, because experienced reviewers read the silences.

The bar moves with risk and intended use

The same body of evidence can pass for one product and fail for another, because the bar is set by what happens if the product is wrong. A tool that suggests a relaxation exercise carries little downside if it errs. A device that decides whether a clinician is alerted to a dangerous result carries a great deal. The higher the harm from error, the more confirmation an assessor requires before accepting the claim.

Intended use is the hinge of the whole assessment, and it is more specific than the technology. The same algorithm can be low risk when it organizes information for a clinician to confirm and high risk when it drives a decision on its own. The claimed use, written down in plain language, determines the evidence required. Widen the claim and the bar rises with it, because the product is now asking to be trusted in situations the narrower claim never covered.

Frameworks differ across jurisdictions, but they share this logic. They sort products into tiers by the consequence of failure and ask for evidence proportionate to it. A clinical prediction model offered as a second opinion is held to a gentler standard than the same model offered to replace human review.

Benefit and risk are weighed as a balance

A product is approved when the expected benefit, judged against the uncertainty that remains, outweighs the expected harm for the use claimed. Assessors accept more residual uncertainty when the benefit is large and the alternatives are poor, and less when the condition is mild and good options already exist.

This is why context changes the verdict. The evidence that satisfies a regulator for a serious disease with no good treatment may fall short for a minor complaint that other products handle well.

What this means for anyone building or reading evidence

If you are assembling a dossier, write the intended use first and let it discipline everything else, because every study will be judged against that sentence. Match your evidence to the population and the decision you are claiming, not to whoever was easiest to enroll. Report the disappointing results alongside the favorable ones, since their absence is the first thing a careful assessor notices.

If you are reading someone else's claim, ask the three questions an assessor asks. Was each study done well enough to believe, does it match the people and the decision you care about, and does the whole body of evidence agree. Then ask what the claim is for and how much harm a wrong answer would cause, because that decides how much proof is enough. A claim that cannot name its intended use has not yet earned a grade. For anything touching your own health, take the specifics to a clinician who knows you.