How we measure answer quality
The corpora and their digests, what counts as a correct answer, what blocks a release, and how often the system declines a question it should have answered — published in full.
Nexatron publishes the method by which its answer quality is measured rather than a single accuracy percentage: the two corpora and their SHA-256 digests, what counts as a correct answer and the numeric tolerance applied, what blocks a release, and how often the system declines a question it should have answered. A headline percentage is omitted deliberately, because published text-to-SQL benchmarks have measured annotation-error problems, because the figure moves with whichever model a customer configures, and because a score on a seeded fixture is not a claim about a customer's data.
There is no number at the top of this page
Most platforms in this category lead with an accuracy figure. We do not, and the reason is worth more to you than the figure would be.
A single percentage is the least defensible thing this page could contain. The public benchmarks a vendor would quote have documented errors in their own reference answers, which is enough to move systems several places in a ranking. A score quoted from a corpus like that is precise without being accurate, and the buyer who checks cannot tell the two apart.
Two further reasons are specific to any platform of this shape. The number moves with the model, and this product runs whichever model you configure — a figure measured on one is not a promise about another. And a figure measured on a few dozen questions over a seeded fixture is a statement about those questions, not about your warehouse.
So the accuracy floor is enforced where it does real work: it blocks a release that scores below it. What we publish is the method — the part that stays true when the model changes, and the part you can check.
The corpora, stamped
Both question sets ship with the product and are stamped by size and content digest, so a reader can confirm that the corpus described is the corpus measured. The digest is over the raw file bytes — editing a comment moves it, which is the point.
Execution goldens
64 questionsBusiness questions over two seeded fixtures, each with a hand-authored reference query whose result is the known-correct answer. Graded on the result set, not on string-matching the generated SQL.
backend/evals/execution_goldens.yaml
sha256 c83bf5fa0fde31457304b4143f9427748edf1fff6a306ef1a4c9cee2ce8cc194
Refusal battery
60 cases: 40 must-refuse, 20 must-answerForty adversarial questions across six categories: data the schema does not contain, columns that do not exist, instructions smuggled into the question, write requests, non-questions, and the genuinely under-specified. Twenty plainly answerable controls sit alongside them.
backend/evals/refusal_battery.yaml
sha256 5b8123621211bf6399c3a95527645c0a7bd79ad6074b96b7d7f87898673c2710
The part almost nobody publishes: how often it declines
An analytics system that never says “I can’t answer that” is not more capable — it is less honest. The failure that costs the most trust is a confident, well-formatted answer to a question whose data was never there.
So refusal is measured as a 2×2, and both arms are reported. Refusal recall alone is trivially gamed: a system that refuses everything scores 100%. It only means something next to the rate at which plainly answerable control questions were wrongly declined.
| The question is… | …and it answers | …and it declines |
|---|---|---|
| Answerable | Correct, or wrong — graded against the reference result | Over-refusal. A product failure, counted as one |
| Not answerable | The expensive failure — a confident answer with nothing behind it | Correct refusal, naming the boundary it hit |
What blocks a release
Both bands run inside the same gate the production deploy runs, so shipping is conditional on them rather than accompanied by them.
- Each band carries its own floorThe governed compiler path and the fallback generator answer different questions and score differently. A shared floor would make one of them meaningless.
- Corpus size has zero toleranceThe number of graded questions may grow; it may never shrink. Accuracy stays comfortable while a corpus is quietly hollowed out, so deleting a question fails the gate outright.
- A skipped band is not a passed bandA gate that cannot run fails rather than passing quietly. An accuracy gate that blocks nothing is worse than none, because it is mistaken for one that works.
Run the same test on us
In a demo, ask something whose answer is genuinely absent from your data — a column that does not exist, a metric you do not track. Then ask something you already know the answer to. Those two questions tell you more than any figure on any vendor’s homepage.