Before a score means anything, the answer key has to be right

The claim

Almost all evaluation effort goes into the model. Almost none goes into the thing the model is graded against.

That asymmetry is harmless until a score starts deciding something — whether a system ships, whether an agent is allowed near a clinical or legal workflow, whether a vendor's accuracy claim survives diligence. At that point the answer key stops being a data file and becomes load-bearing infrastructure. It is usually the one component nobody has checked.

I check it.

Why this is a real failure mode, not a hygiene issue

A wrong key on a capability benchmark mismeasures. A wrong key on a safety benchmark can invert.

If an item's designated correct answer is the action the benchmark was built to discourage, then the unsafe choice scores correct and a model that refuses it scores wrong. A leaderboard containing items like that is partly ranking models by their willingness to do the thing you are testing for. The score doesn't go fuzzy — it points the other way.

Three properties make this class of defect worth specialist attention.

No amount of model-side testing finds it. The defect sits upstream of every model you evaluate. Better models don't surface it; they just score differently against the same wrong target.

It propagates. Benchmarks are increasingly generated from a rule applied to a structured source, rather than written item by item. One flawed rule can seed a defect across hundreds of items, all of which look individually plausible.

Below-chance accuracy is usually read as a fact about models. Sometimes it is a fact about the items. When a whole field of systems performs worse than guessing on a subset, that result should trigger an item audit before it licenses any conclusion about capability.

How the work is done

Six commitments, and each one costs something. That is how you tell them from marketing.

1. The model stays off the measurement path. Labels are re-derived from primary authority — statute, rule of evidence, drug labeling — encoded as executable predicates and applied to each item's own structured fields. The published answer key is never read during derivation; a regression test fails if it is even reachable from the derivation path. Agreement is computed afterwards, as a diagnostic. The result is that no model's mistakes can be confused with the data's, and there is no detection threshold to tune.

2. Census, not sample. Where the precondition holds, every item is checked, not a sampled subset. The counts are exact. There is no confidence interval to argue about and no selection procedure to defend.

3. Evidence is graded, and grades are never summed. A deterministic match, an authored interpretation, a primary-source grounding, and a case held for expert adjudication are four different kinds of thing. They stay four separate records through to the final report.

This is the commitment that actually costs. In one audit it kept a large family of screened items classified as a screen rather than a finding — which is to say I declined to roughly triple the headline number I could have reported, because automatic matching alone supplies no warrant for that promotion. An aggregate count looks equally precise whether or not the evidence behind it is uniform. Keeping the grades apart is the only thing that stops the output of a matcher from becoming a defect count by default.

4. The method's reach is derived, not asserted. Rather than claim general applicability, I classify the entire population against the method's hard precondition and report what fraction it actually reaches — including when the honest answer is a small one. Bounding a method by derivation is less flattering than bounding it by adjective, and considerably more useful to whoever has to decide whether it applies to them.

5. The human is named where the human is load-bearing. Both audits stop at the point where a licensed expert is required, and say so plainly. Additional model readers do not substitute for an attorney or a clinician; they can only map where the disagreement lives and how stable it is. Expert review is not a compliance step appended to the end of the work — it is the mechanism by which the work's ceiling moves. An expert who narrows or widens an encoded rule changes what the check can see, and every number recomputes from the same committed artifacts.

6. Everything that produced a result is attestable. Protocols and analysis plans are hashed before first use. Every model read is recorded with its request and its response. Every read-producing artifact carries the model identifier as served, the harness or runner version, whatever sampling settings the caller can observe, and the timestamp of the write. I work with AI agents on implementation and coding, and the same rule covers them.

That last requirement came out of getting it wrong. An earlier round of reads recorded the harness name and a date — which names a tool, not a model. Establishing which model had actually read what took a forensic recovery from session transcripts on the original host, using records never designed for the purpose. The lesson generalizes past my own work: a role that touches no recorded call path leaves no record of itself. If you are running agent fleets and your audit trail names the runner instead of the model, you do not have an audit trail.

What this is for

Three ways this shows up in client work. They mirror the sprint structure on the home page.

Evaluation Audit — 1–2 weeks. You have an eval suite or a vendor benchmark that a decision depends on. I establish whether its key is sound, whether its items measure what they are named for, and whether any of its results are artifacts of the items rather than the models. You get a graded findings register, not a suspicion ranking.

Gold-Standard Design — 2–4 weeks. You need a keyed evaluation set in a domain where being wrong is expensive. I build it against primary authority with per-item provenance, machine-checkable field semantics, and an explicit statement of the gap between the rule as stated and the doctrine it abbreviates — so that downstream consistency checks can see what they are and are not covering.

Evidence & Provenance Design — 2–4 weeks. You are running AI-assisted review and need a defensible chain of custody from automatic detection to human decision. I design the boundary: what is mechanizable, what is an authored interpretation, what must be adjudicated by a named expert, and what has to be attestable afterwards. This is the part that decides whether your process survives a regulator, an auditor, or a deposition.