AI Confidence Does Not Equal Model Fit
A recent study tested AI systems on numerical estimation tasks, the kind of bounded numerical questions a finance team would put to an AI. When the models reported 99% confidence intervals around their answers, the true value sat inside those intervals only 65% of the time (Epstein et al., FermiEval, 2025). The number on the page said one thing. The actual coverage said something else.
The two numbers carry the same shape, the same percentage notation, the same air of warranted precision. They are not the same kind of object. And the gap matters because finance teams are reading AI confidence figures as if they were the kind of confidence intervals they have trusted for decades.
What the number actually measures
When an LLM generates an answer, it builds the response one token at a time. At each step, it computes scores for every possible next token in its vocabulary and converts those scores into a distribution that sums to 100%. The token with the highest probability is the one selected. The "confidence" the model then reports — whether attached to a number, a forecast, or a confidence interval around an estimate — is derived from these internal probabilities. It is the model's measure of how strongly its chosen output beat the alternatives it considered, on its own internal scale.
In this context, 95% confidence means that the model's internal scoring placed the chosen path well above the alternatives. It does not mean there is 95% confidence that the model outcomes represent the population value. There is no frequency claim, no sampling distribution, no relationship to outcomes in the world. The number describes the model's preference, not the answer's accuracy.
This is why the same model, asked the same question phrased two different ways, can produce two different confidence figures with no change in the underlying truth. The number tracks the model's internal state, not the world.
Imagine an auditor walks in and asks how the number was derived. "The AI was 95% confident" is not a methodology. It is the model's preference score, presented as if it were evidence. The CFO carries that exposure when the confidence figure cannot survive that question.
Why finance reads it wrong
Two mechanisms make this misreading systematic.
The first is pattern recognition turned against the reader. A finance professional's working environment is saturated with numbers that carry warranted meaning. The pattern recognition that makes a finance professional competent is the assumption that a number with this shape comes from a disciplined process. The AI confidence figure borrows the shape without the discipline.
The second is length bias. Research published in Nature Machine Intelligence in 2025 found that users assign higher confidence to longer, more detailed AI explanations, even when the extra length does not improve accuracy. Verbosity reads as evidence of reliability. Auditors are trained to treat thorough documentation as a marker of rigor. LLM verbosity exploits that exact professional habit. The same study identified what the authors call the calibration gap: users systematically overestimate the accuracy of AI responses when those responses come with default explanations.
For finance leaders, these mechanisms compound. The output looks like a confidence interval. The explanation looks thorough. Pattern recognition does the rest. Ownership of the resulting decision sits with the human who signed off, not with the model whose number invited the sign-off. The model does not certify the forecast. You do. The model does not sign the disclosure. You do.
Model Validation
Statistical confidence intervals are trustworthy because of what stands behind them. Construction method, data, assumptions, verification. These are external to any single output. The trust does not live inside the number. It lives in the process that produced it.
AI confidence figures have no equivalent external discipline by default. Model validation is the discipline that builds one.
The core principle is independence. Validation is not a sign-off by the team that built or selected the model. It is a structured challenge by people who did not build it, against criteria set before the model produced its first output, and continued for as long as the model remains in use. Three components carry the load:
- Fitness. Does the model fit the specific decision it is being used for? Vendor accuracy claims are based on vendor test data. A model excellent in its training domain may be unfit for yours.
- Empirical performance. Does it actually perform against ground truth you hold? Running the model against a sample of your own data, with known outcomes, measuring where it agrees with reality and where it does not.
- Vitality. Is it still performing? Models drift as markets, customers, and data sources change. Validation at deployment is not validation forever. The CFO AI Playbook operationalises this as a Monthly Model Vitality Report: continuous monitoring with named owners and escalation thresholds.
Where the ownership sits
The gap between AI capability and AI accountability is a CFO problem whether you know it or not. The confidence number obscures ownership because it appears to come from the model. The consequence of acting on it does not. Whoever signs the report, approves the journal, files the disclosure, or commits the capital owns the outcome. Vendor certification does not transfer that. The model's stated confidence does not absorb it.
Model validation exists to reattach the consequence to a named owner. Independent challenge, documented testing, traceable monitoring. These are the operational forms of ownership. A CFO does not need to understand the inner workings of an LLM to own the result. The CFO needs to refuse the substitution at the front of this article: a number that looks like statistical confidence is not statistical confidence until the methodology behind it has been built and is being maintained.
The number on the output is not the trust. The architecture around it is.
Monday Moves: three actions this week
Inventory the substitution. List every place where AI confidence figures or assurances are currently being read by your team as if they were statistical confidence. Forecasts, risk scores, anomaly flags, classification outputs from vendor tools. The list itself is most of the work, because most organisations have never made it.
Test one output against ground truth you hold. Pick the highest-stakes AI output your finance function relies on. Run it against a sample of your own historical data where you know the answer. Document where it agrees and where it diverges. This is the smallest possible independent challenge, and it is closer to validation than anything most organisations currently have in place.
Name the owner. For every AI-assisted decision in finance, identify by name the human who carries the consequence if the output is wrong. If no name attaches, the decision is not owned. Make continuous validation a condition of use, not a one-time approval at deployment.
