Are Clinicians Well Calibrated? Reading the Evidence on Diagnostic Confidence

Calibration is the match between how confident a clinician feels and how often they are actually right. Studies show the mismatch is real: confidence barely changes as cases get harder, so accuracy can fall sharply while confidence stays high. The dangerous case is not being wrong but being wrong and sure, because high confidence tends to shut down the very rechecking a hard case needs.

Calibration is not the same as accuracy

Accuracy asks how often a clinician reaches the right diagnosis. Calibration asks something subtler: whether their sense of certainty tracks that accuracy. A well-calibrated clinician is confident when they are usually right and appropriately uncertain when they are often wrong, so their feeling of doubt is itself informative.

The two can come apart. A clinician can be reasonably accurate yet poorly calibrated, feeling equally sure on the cases they nail and the cases they miss. It is that disconnect, more than raw error, that makes certainty unreliable as a safety signal.

The finding that should unsettle us

Meyer and colleagues gave general internists a set of case vignettes ranging from easy to hard and recorded both their diagnoses and their confidence. On the easier cases the physicians were right about half the time; on the harder cases accuracy collapsed to a small fraction. Confidence, however, barely moved between the two, staying high even where accuracy had fallen away.

That flat confidence is the heart of the problem. If certainty does not drop when a case becomes genuinely difficult, it cannot serve as an internal alarm. The situations most in need of a second look are exactly the ones where the clinician feels no less sure.

Why overconfidence persists

Berner and Graber, reviewing the evidence, argued that clinicians in general underestimate the chance that their diagnosis is wrong, and that this leans toward overconfidence for reasons both personal and structural. A central structural reason is the feedback gap: clinicians often never learn the true outcome of a diagnosis, so the loop that would correct a miscalibrated sense of certainty rarely closes.

Without reliable feedback, confidence is built on memory of the cases that came back, which are not a fair sample. The result is a stable, self-reinforcing sense of being right that experience does not automatically fix.

Why the confident wrong answer is the dangerous one

An uncertain clinician hedges: they order another test, ask a colleague, or keep the differential open. In the Meyer data, higher confidence was associated with fewer requests for additional testing, which is efficient when the confidence is earned and hazardous when it is not.

The failure mode, then, is specific. It is not simply error; it is error paired with certainty, because certainty removes the behaviors that would have caught the error. This is why calibration, not accuracy alone, is the property that keeps patients safe.

What actually improves calibration

Calibration improves less from being told to be humble and more from structural feedback: systems that report what a diagnosis turned out to be, deliberate follow-up, and reflection on discrepancies. A concrete habit that helps is to ask, before committing, what finding would change the diagnosis and whether that finding was sought.

This reframes doubt as a tool rather than a weakness. A clinician who can state what would prove them wrong, and who later learns whether they were, is being nudged toward the alignment of confidence and accuracy that calibration describes.

Reading a confidence claim

For anyone judging a diagnostic claim, whether from a clinician, a guideline, or an algorithm, confidence should be treated as a separate quantity from correctness and interrogated on its own. The right question is not only how sure the source is but what its track record is on cases like this one, and whether it ever finds out when it was wrong.

An algorithm that reports a probability faces the same test as a person: a high stated confidence means nothing unless it has been checked against how often such claims prove true. Calibration is the discipline of holding certainty accountable to outcomes.