How to verify an AI analytics accuracy claim
7 min readUpdated
An AI analytics accuracy percentage is only meaningful if the vendor also states which corpus it was measured on, which model produced it, what counts as a correct answer, and how the system behaves when the answer is not in the data. A number quoted without those four things cannot be reproduced or compared, because the widely used text-to-SQL benchmarks have measured annotation-error problems and because accuracy moves with the model. Ask for the corpus, the grading tolerance, and the rate of wrong refusals alongside the headline figure.
Why the number on the homepage is usually unfalsifiable
Almost every natural-language analytics vendor publishes an accuracy figure, and almost none publish what it was measured against. That is not necessarily dishonesty. It is that the number is genuinely hard to state responsibly, and the marketing incentive runs the other way.
Two problems make a bare percentage unreliable. The first is the benchmarks themselves. Independent re-annotation of Spider 2.0's reference SQL has reported an error rate on the order of one in ten, enough to move systems several places in a ranking. Re-grading work on BIRD has reported its labels agreeing with human experts only about six times in ten. A score quoted from a corpus with that error rate is precise without being accurate, and a buyer checking the claim cannot tell the difference.
The second is that accuracy is not a property of the product alone. These systems run whichever model you configure. A figure measured on one model is not a promise about another, and a figure measured on a seeded demo fixture is a statement about that fixture. Not about your warehouse, your column names, or your fiscal calendar.
The five questions to ask
- Which corpus, and can I see it?Ask for the question set, its size, and ideally a content hash you can verify. If the corpus is private, the score is not reproducible and should be treated as marketing rather than evidence.
- Which model produced this number?If the platform supports bring-your-own-model, the accuracy figure applies to one configuration. Ask which one, and what the figure is on the model you would actually run.
- What counts as correct?Exact string match on generated SQL is a much weaker test than executing the query and comparing the result set to a hand-authored reference. Ask which one, and what numeric tolerance is applied.
- How often does it refuse when it should answer?A system tuned to never be wrong can simply decline more. Refusal rate and accuracy have to be read together, or a vendor can improve one by quietly degrading the other.
- What happens when the answer is not in the data?This is the question that separates governed systems from fluent ones. Ask to see a live question about a column that does not exist. A confident, well-formatted, entirely invented answer is the failure mode that costs the most trust.
What a verifiable answer looks like
A vendor who can answer those five questions will typically publish the method rather than a single figure. The corpus and its hash, what tolerance is applied when grading, what blocks a release, and how often the system declines a question it should have answered.
Nexatron publishes exactly that, deliberately without a headline accuracy percentage, and explains why on its measurement page. The corpora are committed with the product and stamped by SHA-256 so a reader can confirm that the described corpus is the measured one.
The refusal question, in detail
The most useful evaluation you can run in a vendor demo takes about two minutes and needs no data engineering.
Ask a question whose answer is genuinely absent. A column the schema does not contain, or a metric the business does not track. Then ask a question that smuggles an instruction into it. Then ask something plainly answerable, to check the system has not simply become timid. What you are measuring is whether the boundary is precise: a good system tells you exactly which field it could not see and does not silently substitute a different one.