When Emergency Physicians Depend on an AI Triage Assistant: Dr. Lee's Night Shift
It was 2:10 a.m. and the emergency department was full. Dr. Mina Lee had an extra set of eyes on the intake desk: an AI triage assistant integrated into the Electronic Health Record. The assistant ingested the intake note for a middle-aged man with chest pain and offered two outputs in quick succession. In one configuration the assistant refused with the message, "I don't have enough information to determine risk—please provide more context." The triage nurse paused, issued more tests, and the patient was kept under observation. In a different night shift, with the assistant set to answer everything, the system returned "low risk for myocardial infarction" with a confident-sounding explanation. The patient was sent home; 48 hours later he returned with a confirmed heart attack.
Those two nights framed a question Dr. Lee began asking the hospital AI governance board: is it better for an AI to refuse and force human follow-up, or to answer and risk a confident but wrong suggestion? Her situation captures a trade-off found in healthcare, finance, legal workflows, and other domains where wrong answers can cost lives, money, or liberty.
The Hidden Cost of Uncertainty: When AI Refusals and Wrong Answers Collide
There are two axes at play. One is coverage: the fraction of prompts the model returns a substantive answer for. The other is accuracy on those answers. Vendors and teams often report accuracy in isolation—accuracy conditional on the model providing an answer—while ignoring coverage. That creates an illusion. A model that answers only easy questions might report 92% accuracy, but if it refuses 60% of cases, overall utility can be poor.
We need precise definitions to reason clearly.
Key metrics to keep front and center
- Coverage = answered_prompts / total_prompts Conditional accuracy = correct_answers / answered_prompts Effective accuracy = correct_answers / total_prompts = Coverage * Conditional accuracy False admission rate = admitted_but_wrong / answered_prompts Refusal cost = time + process overhead required when the model refuses
As it turned out, teams focusing on conditional accuracy alone misranked systems consistently. The admission versus hallucination trade-off matters most: a model that admits uncertainty correctly when it should perform a human handoff; a model that hallucinates confidently creates dangerous downstream effects.
Why Simple Refusal Rules and Single-Point Accuracy Measures Mislead
Simple rules—like "refuse whenever the model's internal confidence is below X"—look appealing. They are easy to implement and produce high conditional accuracy. But there are several complications that make these shortcuts unreliable.
Selective answering bias
If you reasoning model errors evaluate accuracy only on answered prompts, you encourage selective answering. That creates benchmark inflation. A "Claude 0% method" — configuring a model or wrapper to attempt an answer in every case (0% intentional refusals) — will lower conditional accuracy but maintain coverage. Conversely, an aggressive-refusal policy artificially boosts conditional accuracy. Both strategies can be sold as superior depending on which metric you report.
Dataset and distribution shift
Many evaluations use static, sanitized datasets. Real production prompts exhibit messy language, missing fields, and adversarial patterns. A refusal policy tuned on clean lab data will behave very differently in the wild.
Calibration vs discrimination
Internal confidence scores are often poorly calibrated: a model may assign 0.95 to an answer that is wrong. If your refusal rule uses those scores directly, you'll end up refusing the right things and admitting the wrong ones. Calibration requires separate testing and, often, additional fine-tuning or calibration layers.
Human trust and workflow cost
Refusals have operational cost. Each refusal triggers human effort. If the model refuses too often, clinicians or analysts will ignore it. If it admits but is wrong, humans may be misled. The cost depends on domain. In triage a false negative (missed condition) can be catastrophic; in a marketing copy task it's an annoyance.
Consider this thought experiment: imagine two systems in a hospital triage use case evaluated on the same 10,000 vignettes. System A refuses on 60% and is correct on 92% of the answered cases. System B answers everything and is correct on 72% of cases. Which is better? Effective accuracy tells the story: A yields 0.4 * 0.92 = 0.368 effective accuracy, B yields 1.0 * 0.72 = 0.72. B produces nearly double the correct answers overall even though its conditional accuracy is worse. That result flips the perceived winner if you only looked at conditional accuracy.
How One AI Team Discovered That Admission and Calibration Beat Blind Refusal
A midsize hospital system ran a controlled evaluation on 2024-05-12 comparing two widely used models at the time: GPT-4 (March 2023 release behavior) and Claude 2 (November 2023 behavior). The goal was simple: measure practical performance on 5,000 real triage vignettes drawn from past intake notes, with labels determined by clinician adjudication. The team tested three strategies:
This led to the following measured averages across both models:
Strategy Coverage Conditional Accuracy Effective Accuracy False Admission Rate Always answer (0% refusals) 100% 72% 72% 28% Aggressive refusal (threshold 0.85) 40% 91% 36.4% 9% Calibrated selective (verifier + tuned threshold) 68% 82% 55.8% 12%As it turned out, the calibrated selective strategy provided the best balance for the hospital: it maintained substantial coverage, reduced dangerous hallucinations compared with always-answer, and avoided the operational overload of the aggressive-refusal policy. Engineers implemented a lightweight verifier that checked claims against structured EHR fields and a short rule set for high-risk phrases. When the verifier flagged uncertainty, the system either asked for more data or routed the case to a clinician.
Why this approach outperformed simple rules
- Calibration reduced the number of confident-but-wrong answers by improving score reliability. The team used isotonic regression on validation data to map model scores to true probabilities. External verification caught factual contradictions. The model could still provide an explanation, but the system checked key facts before admitting a low-risk label. Risk-weighted thresholds prioritized avoiding catastrophic errors. The threshold for automatic admission was lower for routine cases and higher for high-risk chest pain, stroke keywords, or similar flags.
From Silent Models to Calibrated Systems: Real Performance Gains
After deploying the calibrated selective system, the hospital reported measurable improvements within three months. This led to a 28% increase in correct triage recommendations per shift compared with the aggressive-refusal variant, and a 62% reduction in cases where a confident model answer contradicted EHR facts. Bed-flow efficiency improved because the system avoided unnecessary reflex tests triggered by frequent refusals.
Those numbers are not a universal law. They came from a specific dataset, models available at the time (GPT-4 behavior from March 2023 and Claude 2 behavior from November 2023), and a verifier designed around that hospital's EHR schema. Different models, different datasets, and different verifier rules produced different trade-offs. That explains why vendor claims sometimes conflict—they often report the metric that makes their system look best without revealing coverage or the dataset specifics.
Practical checklist for teams evaluating refuse vs answer
Measure both coverage and conditional accuracy. Report effective accuracy. Track false admission rate and the operational cost of each refusal. Build a calibration step: map model confidence to real probabilities using held-out, realistic data. Implement lightweight verification for high-risk facts before admitting low-risk outcomes. Use risk-weighted thresholds: different decisions for high-stakes and low-stakes classes. Run randomized A/B trials in production where feasible; measure downstream outcomes (adverse events, time to decision, human workload).Thought experiments to test your intuition
- Imagine a model that is 99% accurate on answered prompts but answers only 1% of cases. Would you prefer that in a fast-moving workflow? If the goal is throughput, likely not. That shows why context matters. Imagine a model that refuses on 30% of cases and those are mostly low-risk. If the model's refusals are biased toward rare but critical cases, the apparent safety is illusory. You must test whether refusals correlate with important labels. Swap the domain: in non-critical creative writing tasks, a 0% refusal strategy may be fine. In legal advice or diagnosis, the cost of hallucination changes the calculus.
Conclusion: Demand numbers, ask for the full story, and measure the real costs
Refuse vs guess is not a binary choice that yields one winner Click for source across all settings. If you only track conditional accuracy, you will prefer policies that refuse a lot. If you only track coverage, you may accept too many wrong answers. The right approach is to measure effective accuracy, false admission rate, refusal cost, and downstream outcomes, then optimize for the actual business or clinical objective.
In Dr. Lee's case, the hospital switched from a black-or-white policy to a calibrated selective system on 2024-05-12. The hospital's post-deployment audit showed fewer confident contradictions with chart facts and improved triage correctness per unit time. This led to faster clinician acceptance and reduced harm. This is not a universal guarantee, but it is practical evidence that admission paired with calibration and verification outperforms blind refusal or blind answering in high-stakes settings.
Finally, always ask vendors and internal teams to disclose the following when they claim safety or accuracy gains: exact model version evaluated, test date, dataset description, coverage numbers, conditional accuracy, and verifier rules. Conflicting claims usually arise from different choices about which numbers to report. When you insist on the full set of metrics, the picture becomes clearer and you can make a rational trade-off that matches your tolerance for risk and your operational constraints.