← Vociferous AI · Research · Medicine
When a safety benchmark infers safety from silence
Scope, stated first
This describes a research result and a capability. It is not a medical product, it is not medical advice, and it does not describe a clinical client engagement. Any use that touches real care keeps a qualified clinician in charge. My training is a Ph.D. in Chemical Engineering from MIT, in the MIT Biotechnology Center — I am not a clinician, and the work is built so that the clinician's judgment is the thing that moves its ceiling.
The audit
I ran a structural audit of a public medication-safety benchmark whose documented construction rule infers that a drug is safe from the absence of a recorded risk.
That inference is the entire issue. Absence of evidence in a source database is not evidence of absence in a patient. The rule is applied mechanically across the benchmark, which means one flawed premise seeds a defect across many items, each of which looks individually plausible to a reviewer spot-checking a sample.
The audit shows what that inference does not guarantee, measures how often the gap is actually realized across the whole benchmark rather than a sampled subset, and proposes a quarantine-and-review workflow that separates what a maintainer can fix mechanically from what requires a clinician. The paper is under peer review; title, venue, and full findings stay private until the decision lands.
Why a safety benchmark is a different problem
A wrong key on a capability benchmark mismeasures. A wrong key on a safety benchmark can invert.
If an item's designated correct answer is the action the benchmark was built to discourage, then the unsafe choice scores correct and a model that refuses it scores wrong. A leaderboard containing items like that is partly ranking models by their willingness to do the thing you are testing for. The score does not go fuzzy — it points the other way.
Three properties make this worth specialist attention. No amount of model-side testing finds it, because the defect sits upstream of every model you evaluate. It propagates, because benchmarks are increasingly generated from a rule applied to a structured source rather than authored item by item. And below-chance accuracy is usually read as a fact about models when it is sometimes a fact about the items — when a whole field of systems performs worse than guessing on a subset, that should trigger an item audit before it licenses any conclusion about capability.
What I build
Source-grounded literature synthesis. Retrieval and synthesis grounded in the actual literature, where every claim ties back to a paper, guideline, or source a reader can open. Conflicting evidence is shown as conflicting rather than averaged away, and confidence is stated plainly. The system organizes the evidence; it does not pretend to settle it.
What it is good for. Literature reviews and treatment-option comparisons, preparation for a specialist consultation, and second-opinion research support — the work of gathering and structuring evidence so a person can reason about it well. It is strongest as preparation for a human decision, never as the decision.
Where it must not go. It does not diagnose, it does not prescribe, and it does not replace a clinician. On health questions the cost of false confidence is measured in harm, so the system is built to defer: surface the evidence, mark the uncertainty, and route the judgment to a qualified human.
What this is worth to a team
You maintain a clinical benchmark or eval set. The question is whether its construction rule states an inference it cannot support, and how many items inherit that. A census answers it exactly; a sample does not.
You are buying a clinical AI tool on the strength of a score. Ask what the score was computed against, whether the key was derived independently of any model, and what the system does when it should decline to answer. Those three questions separate most vendors.
You are building, and the safety behavior is the product. Then abstention, calibrated confidence, and a named point where a clinician takes over are not features to add later — they are the architecture.
The method behind all of it — model off the measurement path, census rather than sample, evidence grades never summed, the human named where the human is load-bearing — is on the Research page.
If this is your field
If you work in health, clinical research, or medical knowledge tools and want AI that stays honest about what it knows — or you maintain an evaluation set that a clinical decision depends on — send the workflow.