Why 'our AI is 95% accurate' tells you almost nothing — and what to ask instead

Every AI vendor has an accuracy number. It is usually a high one. It is usually in a marketing document rather than a technical paper. And it usually tells you almost nothing about whether the model will perform reliably in your specific context. Here is why — and here are the five questions that cut through it.

What an accuracy claim hides

The base rate problem. A model that classifies loan applications and approves 90% of them could achieve 90% accuracy by approving every single application and never denying anyone. If 90% of applicants would repay their loan regardless of any model, the model has "90% accuracy" while contributing nothing and potentially creating a fair-lending liability by the pattern of who falls in the remaining 10%. Accuracy without a base rate is a meaningless number.

The test set problem. Models are evaluated on a held-out portion of the data they were trained on. If the training data over-represents certain demographics, geographies, or time periods, the test set carries the same bias — and the model can score highly on the test set while performing very differently on production data. "95% accurate on our test set" means "95% accurate on a subset of our historical data" — which is a much weaker claim.

The ground truth problem. For many consequential decisions, there is no clean ground truth. A credit model evaluated against "repaid the loan" as ground truth is actually evaluating its own influence — applications the model approved are the only ones that got a loan, so the only repayment data available is for the people the model already decided were creditworthy. This circularity makes the model's accuracy look better than it is, because the model can never be wrong about the applicants it denied.

The drift problem. A model trained on 2022 data, evaluated on 2022 data, and claimed to be "95% accurate" may behave very differently on 2025 data if the underlying patterns have shifted. Economic conditions, borrower behaviour, regulatory context, and data distributions all change. Without a live monitoring process, an accuracy claim is a point-in-time measure, not an ongoing guarantee.

The sub-group problem. Aggregate accuracy can mask disparate performance across demographic groups. A diagnostic AI that is "90% accurate" might be 95% accurate for one demographic and 75% accurate for another. If the lower-performing group is already a medically underserved population, that performance gap is a patient harm and a regulatory liability — but it is invisible in the headline accuracy number.

The five questions every buyer should ask

1. What is your test set, and how was it constructed? Is it drawn from the same distribution as your training data, or was it constructed to represent the production environment? How was data from time periods used to avoid look-ahead bias?

2. What is your ground truth definition, and does it avoid circularity? For decision-making models, how was ground truth established for the cases the model would have rejected? Independent human review? Randomised approval trials? This question reveals whether the accuracy claim can even in principle be meaningful.

3. What is the base rate, and how does accuracy compare to a naive baseline? What would a model that always predicted the majority class achieve? What would a simple rule-based system achieve? A model that barely outperforms a baseline is not providing the value the headline number implies.

4. What is the accuracy by sub-group for the populations affected by this model? If the vendor cannot produce sub-group performance breakdowns, you cannot evaluate whether the model performs equitably across the people whose decisions it influences. For regulated applications, this is not a nice-to-have; it is a legal requirement in most jurisdictions that have enacted AI-specific rules.

5. What is your monitoring process, and what would trigger a model review? A vendor who can name specific performance thresholds and a documented process for what happens when those thresholds are crossed has an actual accuracy assurance process. A vendor who says "we monitor continuously" without specifying what that means does not.

How an iDharma audit independently verifies accuracy claims

When we audit a model's accuracy, we do not accept the vendor's test results. We request the test data, the evaluation methodology, and the ground truth construction process, and we evaluate them independently against the five criteria above. We also request sub-group performance data for every protected characteristic relevant to the deployment context.

For models where the vendor cannot or will not provide this documentation — which itself is a finding — we document the gap and rate it as a high-severity risk. A model whose accuracy cannot be independently verified is a model whose accuracy you cannot rely on in a regulatory examination or a litigation context. That is not a technicality; it is the difference between a defensible deployment and an exposed one.

If you have received accuracy claims from a vendor and want them independently evaluated, request an audit. If you want to understand your exposure before making that decision, start with the free Risk Snapshot.