How to Judge an FDA Cleared Radiology AI Tool

How to Judge an FDA Cleared Radiology AI Tool

An FDA clearance tells you a radiology AI tool reached the market legally. It does not tell you the tool was proven to help patients. Most imaging AI is cleared through the 510(k) pathway, which asks whether a new device is substantially equivalent to one already on the market, not whether it improves diagnosis or clinical outcomes. To judge a tool honestly, read past the clearance stamp and ask three questions: what was tested, on whom, and how were the reported sensitivity, specificity, and area under the curve produced.

What a 510(k) clearance actually certifies

The FDA maintains a public list of AI-enabled medical devices, and radiology accounts for roughly three quarters of the entries, by far the largest category. The great majority of these tools enter the market through premarket notification, better known as 510(k). Under this pathway, a manufacturer demonstrates that its device is "substantially equivalent" to a legally marketed predicate device, meaning it has the same intended use and either the same technological characteristics or differences that do not raise new questions of safety and effectiveness. That is the FDA's own language, and the distinction matters: substantial equivalence is a comparison to an existing product, not independent proof that the software makes radiologists more accurate or patients better off.

In the analysis by Khunte and colleagues, published as a medRxiv preprint and later in Clinical Radiology, 148 of 151 imaging AI algorithms cleared through November 2021 went through 510(k), and only three used the De Novo route reserved for genuinely novel devices. Almost none faced the premarket approval studies expected of high-risk hardware. So a clearance letter confirms that a regulatory comparison was accepted. It is the floor of credibility, not the ceiling.

Reading sensitivity, specificity, and AUC

Three numbers dominate imaging AI summaries, and each answers a narrow question. Sensitivity is the share of truly abnormal cases the tool flags; a highly sensitive tool misses few cancers but may raise many false alarms. Specificity is the share of truly normal cases it correctly leaves alone; a highly specific tool avoids false alarms but can miss subtle disease. The two trade off against each other, and a vendor can tune the operating threshold to make either look impressive in isolation.

Area under the curve, or AUC, summarizes that trade-off across every possible threshold. An AUC of 0.5 is a coin flip and 1.0 is perfect, so higher discrimination looks reassuring. But AUC says nothing about the threshold a hospital will actually use, and it hides how the tool behaves at the disease prevalence of your population. A model validated on an enriched dataset where half the scans are abnormal will generate far more false positives in a screening clinic where fewer than one percent are. Positive predictive value, the number a clinician feels day to day, falls sharply as prevalence drops even when sensitivity and specificity stay fixed. Ask, too, whether the reported figures come from the algorithm alone or from radiologists reading with and without it, because the second design is the one that tells you whether the tool actually changes human decisions.

The gaps the clearance summaries document

The published record shows where the evidence thins out. Across those 151 imaging algorithms, 97, or about 64 percent, reported using clinical data to validate the device at all, which means roughly a third described no clinical validation in their public summary. Multi-site testing was the exception rather than the rule: only 51, about 34 percent, characterized their validation data as multicenter, while most did not specify. Much of the testing was retrospective, run on stored images rather than in live workflow.

The transparency gaps are starker for generalizability. Only six of the 151 summaries, about four percent, disclosed the demographic makeup of the study population, and only around five percent reported the scanner models used. A tool can perform well on one vendor's CT scanner at one field strength in one demographic and quietly degrade on different equipment or a different patient mix. When that information is missing, you cannot tell whether a published AUC will hold in your department. This is educational information and not medical advice, but the pattern is a good reason to treat vendor performance claims as hypotheses to test locally rather than guarantees to accept.

A short checklist

When a tool crosses your desk, work through a few questions before trusting the marketing deck:

• Was the clearance based on standalone algorithm metrics or on radiologists reading with the tool? Only the second speaks to clinical impact.
• Retrospective or prospective, single-site or multi-site? Prospective, multi-reader, multi-site testing is stronger and still uncommon.
• Does the validation population resemble yours in age, sex, disease prevalence, and scanner vendor? Silence on these points is itself a finding.
• At what prevalence were the numbers generated, and what does positive predictive value become at your prevalence?
• What is the predicate, and did the cleared tool inherit its claims rather than earn new ones?

None of this makes FDA-cleared imaging AI untrustworthy. It means clearance and clinical proof sit at different thresholds, and the FDA's public list plus the peer-reviewed audits of it give you enough to tell where a given product falls between them. The strongest tools will welcome these questions; the weakest will answer with a clearance number and little else.