Why we do not publish an accuracy number

August 11, 20266 min readFor anyone evaluating an AI analytics platform

Accuracy percentages for natural-language analytics are difficult to compare because they depend on which question set was measured, which model produced the answers, and what counted as correct. The widely used public benchmarks have documented labelling errors of their own, so a score drawn from them can be precise without being accurate. Publishing the measurement method, the question sets and the refusal rate gives a buyer something they can check.

The number would have been easy to produce

We run a set of questions with known answers against the product on every release, and it blocks the release if it scores below a floor. Turning that into a marketing percentage would have taken an afternoon.

We did not, and the reason is worth more to a buyer than the figure would have been.

Three reasons the figure would mislead

The public benchmarks have their own errorsIndependent re-checking of the reference answers in the widely used industry test sets has found error rates high enough to move a product several places in a ranking. A score drawn from a flawed answer key is precise without being accurate, and the buyer checking it cannot tell the difference.
The number moves with the modelThis product runs whichever AI model a customer configures. A figure measured on one is not a promise about another, so quoting it without naming the model is quoting a number about a configuration you may never run.
It is a statement about our test data, not yoursA few dozen questions over a demo dataset says something about those questions. Presented as "our accuracy", it becomes a claim about your business, your column names and your financial calendar, which nobody has measured.

What we publish instead

The question sets, with a fingerprint you can verify so you know the set described is the set measured. What counts as a correct answer and how close is close enough. What blocks a release. And the part almost nobody publishes: how often the system declines a question it should have answered.

That last one matters because refusal rates and accuracy have to be read together. A product tuned never to be wrong can simply refuse more, and its accuracy score goes up while it becomes less useful. Reported alone, either number can be improved by quietly degrading the other.

What to do with this

Ask every vendor on your list three questions: measured on which question set, using which model, and how often does it wrongly refuse. A vendor who can answer all three has done the work. One who cannot has a number from a slide.

You can run a better test yourself in about two minutes. Ask something you already know the answer to, then ask something your data cannot answer, and watch which of those two the product handles badly.

Three reasons the figure would mislead

The public benchmarks have their own errorsIndependent re-checking of the reference answers in the widely used industry test sets has found error rates high enough to move a product several places in a ranking. A score drawn from a flawed answer key is precise without being accurate, and the buyer checking it cannot tell the difference.

The number moves with the modelThis product runs whichever AI model a customer configures. A figure measured on one is not a promise about another, so quoting it without naming the model is quoting a number about a configuration you may never run.

It is a statement about our test data, not yoursA few dozen questions over a demo dataset says something about those questions. Presented as "our accuracy", it becomes a claim about your business, your column names and your financial calendar, which nobody has measured.

What we publish instead

What to do with this