Measuring AI on financial language, and then making it better

Four strands

Finance runs through most of what I've built, in four different forms.

Infrastructure for financial services. I co-founded Socure and served as its CTO — a digital identity-verification company whose business is deciding, at speed and at scale, whether the person on the other side of an account opening is who they claim to be. I built the early technical case and took the first meetings with American Express and other major financial institutions — which is where you learn quickly that an institutional buyer does not evaluate a model, they evaluate whether your claim about the model will survive their risk, compliance, and procurement review.

That is a machine-learning product operating inside a regulated workflow, where a wrong answer is a chargeback, a declined good customer, or a compliance finding. It is also where I first had to answer the question this page is about: what exactly was this measured against, and who checked that.

Enterprise delivery for financial institutions. From 2013 to 2021 I was the long-term consulting lead at Sia/Addison, supporting complex digital, technical, and security-sensitive engagements for Fortune 50 companies, major financial institutions, and leading law firms. I led architecture, application development, technical strategy, and cybersecurity-related work, and worked directly with senior stakeholders to translate business goals into buildable systems — across multiple years of repeat engagements in environments where reliability, security, and execution mattered. Client names and matters stay confidential.

Consumer fintech venture advisory. I served as strategic advisor to the founding team of an early-stage consumer fintech, running the full arc that turns an idea into an investable plan for a new consumer payments product. The client and product are confidential; the shape of the work is below. Across my other ventures I do the market, pricing, licensing, and unit-economics work alongside the engineering rather than handing it off, which is why technical choices stay attached to a cost and a business case instead of drifting from one.

Measuring finance AI honestly, and then making it better. This is the current work, and it has two halves that belong together: auditing whether an evaluation of financial language means what it claims, and developing methods that raise what a model can actually deliver on financial source documents. The rest of this page is about both.

The primitive nobody audits

A large share of applied finance AI reduces to one question: given this passage from a filing, an earnings call, or an annual report, does this claim follow from it, contradict it, or neither?

That is natural language inference, and it sits underneath the things people actually want. Does the summary say anything the document doesn't support. Does this quarter's guidance contradict last quarter's. Does the risk factor actually state the exposure the analyst has attributed to it. Is the number in the narrative the number in the table. Every one of those is an entailment judgment wearing a product name.

So when a vendor or a team reports that their system is 78% accurate on financial inference, that number is load-bearing. It decides whether the system goes near an analyst's workflow. And it is produced by grading the model against an answer key — which is, almost always, the one component in the pipeline nobody has checked.

My current research checks it: a full independent reproduction of published model baselines on financial inference, run end to end on my own hardware, against an evaluation set carrying multiple finance-professional judgments per item rather than a single label — with calibration and error-overlap diagnostics across several model families, from zero-shot general-domain classifiers through a domain fine-tune to a 70B open model.

That reproduction is the foundation for the second half of the work, which is building methods that improve what a model gets right on financial source documents. It is in progress. The dataset, the paper it audits, and the specific findings stay unnamed here until the work is finished.

What I can describe now is the failure modes, because they are not specific to one dataset. I have found some version of each of these more than once.

Five ways a finance AI score misleads you

1. The model may not be reading the document. Withhold the source passage entirely and give the model only the claim it is supposed to verify. If it still scores well above chance, then some of its reported accuracy is coming from the phrasing of the claim — length, hedging, negation, how a generated statement tends to be worded — and not from the filing. This is cheap to test and rarely tested. It does not mean the system is worthless; it means the headline number is partly measuring something other than financial reading comprehension, and you should know which part before you trust it on a document it has never seen.

2. One gold label hides a panel of experts who disagreed. Financial judgment is not uniformly crisp. When several finance professionals label the same item, a substantial minority of items are contested — decided by majority, often with at least one annotator marking low confidence. Collapsing that into a single right answer throws away the most useful signal in the whole dataset, which is where the experts themselves split. Models err disproportionately on exactly those items. A system that is 95% accurate on the unanimous items and near chance on the contested ones is a very different product from one that is uniformly mediocre — same headline score, opposite deployment decision. Ask for the breakdown.

3. The cases the model couldn't answer may have been deleted from the score. This one is mechanical and I find it in evaluation harnesses repeatedly. When a model returns something unparseable — a refusal, a malformed label, an empty completion — a scoring routine that only counts predictions inside the valid label set silently drops that item from the confusion matrix. The reported accuracy is then computed over the subset the model managed to handle. In a small worked example, three correct answers and one unparsed output scored 100%; counting the failure honestly gave 88.9%. In production that gap is the difference between a system that works and one that works when it works. A finance evaluation needs an explicit policy for abstention and failure, and abstentions belong in the denominator or in their own line — never in neither.

4. Train and test may share more context than the split implies. Benchmarks built by generating many claims from a pool of source documents can put the same underlying passage on both sides of the split, with different claims attached. That is not duplicate data, and it usually passes a duplicate check — but the model has seen the document. Performance on those items is systematically better, and the overall score therefore overstates how the system will do on a filing it has genuinely never encountered. The fix is to report the clean subset separately, and to group splits by source document rather than by item.

5. "Best model" often means "best model we were allowed to run." Published finance benchmarks frequently exclude frontier commercial models, for reasons that are entirely legitimate — data governance restrictions on sending institutional text to external APIs, output-licensing terms, a preference for reproducible open weights. The reason is rarely stated. The consequence is that a leaderboard's top number is the best open result, not the state of the art. If your institution has a data-governance posture that permits something the benchmark authors could not use, the published ceiling is not your ceiling. If it doesn't, the published ceiling may be exactly right — and that is worth knowing deliberately rather than by accident.

The other half: methods that raise the output

An audit tells you a number is wrong. That is necessary, and on its own it is not worth much to anyone with a product to ship.

The useful part is that the artifacts a rigorous audit produces are exactly the inputs a better method needs. Once you have per-item predictions and class probabilities from several model families on the same financial corpus, plus a panel of expert judgments with confidence rather than a single flattened label, you are holding the raw material for improvement rather than just a verdict. Five directions I am developing on that basis.

Domain fine-tuning, and the label noise that caps it. General-purpose inference models transfer to financial language poorly — the vocabulary of guidance, accrual, restatement and hedged forward-looking commitment is not the vocabulary they were trained on. Fine-tuning on in-domain financial text recovers real ground, and I run that end to end rather than quoting someone else's result. The less obvious finding is that the ceiling is usually not the model. Training sets for financial inference are commonly generated by a large model and never checked by a human, so a meaningful fraction of training labels are simply wrong. Cleaning the training labels is frequently a larger lever than a larger model, and far cheaper.

Combining models that fail differently. Across model families on the same financial items, the errors turn out to be substantially complementary — the items a fine-tuned domain model gets wrong are not the items a large general model gets wrong. That is the precondition for a combiner or a router that picks per item rather than globally, and it is the method I am building next. I am holding it in advance to a standard that most ensemble work skips: it has to beat the best single model and simple fixed combinations, it has to be fit on data separated from the data it is evaluated on, and the oracle number gets reported as headroom rather than as a result. Until that clears, the honest claim is complementarity measured, not accuracy delivered.

Training toward the expert distribution instead of one answer. When three finance professionals label an item and split two-to-one, the majority label is an impoverished summary of what actually happened. Using the vote distribution and the annotators' own confidence as the training target teaches the model something a hard label cannot: that some financial judgments are genuinely contested, and roughly how contested. A system that knows the difference between a clear entailment and a close call is more useful in a workflow than one that is slightly more accurate on average.

Calibration and selective prediction. In a finance workflow the metric that matters is almost never raw accuracy. It is accuracy at a coverage level you choose — how good the system is on the 70% of items it is willing to answer, with the remaining 30% routed to a person. That only works if the model's confidence means something, which by default it does not. Calibrating it, then setting the abstention threshold against the real cost of a wrong answer versus the real cost of a human review, converts a benchmark score into an operating decision. This is also where an audit and a method meet: the failure mode where unanswerable items vanish from the score is the same mechanism that makes a well-designed abstention policy look like a loss.

An evaluation protocol that does not fool its author. None of the above is believable without it. Split by source document rather than by item, so the model cannot be rewarded for having seen the filing. Fix the estimand before tuning — majority-label accuracy, agreement with the expert distribution, calibration, and selective prediction are four different objectives and collapsing them into one hides the trade-off you are making. Keep anything you fit strictly separate from anything you report. Say plainly when a set has already informed exploratory analysis, because it has stopped being a clean test.

These are in development. Until gains are measured under that protocol, I report the design and the diagnostics — not the gains.

What this is worth to a finance team

Four situations where this stops being academic.

You are buying. A vendor's accuracy claim is a number produced by a procedure. Diligence usually interrogates the model and accepts the procedure. The five failure modes above are all invisible to model-side testing, all survive a competent technical review, and all inflate the number in your favor — which is to say, in the vendor's.

You are shipping. If an internal system is about to sit next to an analyst, a trader, or a client-facing disclosure, the useful question is not its average accuracy. It is where the accuracy is concentrated, what it does when it cannot answer, and whether anyone can reconstruct afterwards which model produced which output. That last one is an audit-trail problem, and it is usually discovered at the worst possible time. If your logs name the harness or the agent runner instead of the model as served, you do not have an audit trail.

You have to defend it. Model risk, internal audit, and regulators all eventually ask the same thing: what was this validated against, and who checked that. An evaluation whose answer key has never been independently derived is a weak answer to a question that gets asked under pressure.

You are building, and you have hit a ceiling. A system reads your filings, earnings transcripts, or credit memos, and it is good but not good enough, and a bigger model has stopped helping. That is usually a data and protocol problem rather than a model problem: noisy training labels, a hard gold label where the experts actually disagreed, uncalibrated confidence that makes abstention impossible, or a test split that has been quietly flattering you. Those are fixable, and fixing them is cheaper than the next model tier.