LLM evaluation metrics for analytics: what to measure when the answer is a number

Generic text metrics grade the wrong object

Most lists of LLM evaluation metrics were written for summaries and chatbots, so they score similarity, relevance and tone. An analytics answer can be checked. Say net revenue for the quarter was 4.2 million. A beautifully worded 4.6 million is a failure, and a similarity score may barely notice, because the two sentences differ by one digit.

OpenAI’s evaluation guidance lists reliance on generic academic metrics such as perplexity and BLEU among its anti-patterns, along with judging by whether something seems to work. Anthropic’s guidance starts in the same place and asks for evals that mirror the real task distribution. I’d take that to its conclusion for analytics and order the metrics by what can hurt the reader, with the deterministic grades at the bottom and the softer ones above.

Nine metrics, one question each

Each metric below answers one question and has one natural way to compute it. Where a metric needs a reference answer, that reference comes from the person who owns the number. The sections after the table cover the parts teams get wrong.

Grade the result set, and define equality first

Comparing generated SQL with reference SQL as text is a poor metric, because many different queries are correct. Run both and compare what they return. The BIRD benchmark made this style of evaluation familiar for large databases with messy real values, and I’d make it the default for your own set too.

The hard work is deciding what equal means, and that decision should be on paper before any candidate runs. Row order only matters when the question asks for a ranking. Column names differ harmlessly. Floating point needs a tolerance, small enough that it can’t absorb a real definitional difference. Extra columns are usually fine and missing ones aren’t. Treat an empty result that matches an empty reference with suspicion, since a bad filter produces the same thing, and keep a few cases whose reference is known to be empty for a stated reason.

The check below works in most warehouses and handles the common case. Run the comparison in both directions, and round first so the tolerance is explicit in the query.

-- Rows the candidate returned that the reference did not
SELECT region, ROUND(net_revenue, 2) FROM candidate_result
EXCEPT
SELECT region, ROUND(net_revenue, 2) FROM reference_result;

-- Rows the reference has that the candidate missed
SELECT region, ROUND(net_revenue, 2) FROM reference_result
EXCEPT
SELECT region, ROUND(net_revenue, 2) FROM candidate_result;

-- Pass only if both queries return zero rows

Score declining in both directions

A system can raise its correctness score by refusing anything difficult, and it can raise its coverage by inventing answers. You need a two-way score to see both. Label each case answerable or unanswerable in advance, then count all four outcomes.

Take a synthetic set of 200 cases, 140 answerable and 60 unanswerable, where the system declines 7 answerable questions and answers 9 unanswerable ones. Its over-refusal rate is 7 of 140, or 5 percent. Its false-answer rate is 9 of 60, or 15 percent. Those are separate problems with separate fixes. The first usually points to missing context or an overcautious rule, the second to a system that can’t tell when its sources don’t contain the answer.

Ambiguity gets its own label. Where a measure has two meanings that return different numbers, the system should ask or state which meaning it used. Where the meanings agree, a clarifying question only wastes the reader’s time, so count it against the system.

Permissions are a count against zero

Permission failures should never enter an average. Create test identities with known scope, ask the same questions as each, and compare what comes back with what that identity may see in the source. One leaked row fails the run, however good everything else looks.

Don’t overstate what zero failures proves. The rule of three says that with no events in n trials, the 95 percent upper bound on the true rate is about 3/n. Passing 150 permission cases supports a claim that the failure rate is below about 2 percent, a long way from never. I would enforce identity at the source system, test it as well, and treat the evaluation as a check on the design.

Check the narrative in code before asking a judge

The right table can still be followed by a wrong sentence, with a percentage computed in the model’s head or a cause nothing supports. Numeric grounding catches the first kind cheaply. Extract every number in the prose and require each to appear in the result, or to be derivable from it by a calculation the system recorded. A parser does this, so it costs little and returns the same grade every run.

The Ragas metrics, including faithfulness and response relevancy, target the same concern for retrieved text and were designed to work without ground truth annotations. They help with unstructured context and with production traffic where no reference exists. For a claim that a number is right, they’re a supplement, because a faithful summary of the wrong query is still wrong. Ragas also lists SQL metrics, including an execution-based comparison, and those fit the result-set grading described earlier.

Causal claims need a person or a judge model, and a judge needs calibrating first. OpenAI’s guidance recommends checking LLM judges against human labels, warns that judges can favor longer responses and be swayed by the order they see them in, and suggests pass or fail grading and pairwise comparison. Anthropic’s advises grading with a different model from the one that produced the output. I’d have two people grade 50 outputs against the rubric, fix the rubric where they disagree, then measure the judge against the settled labels, and trust it only at the agreement you decided in advance.

Put an interval on every score

A score from a small set is a range. Suppose a system gets 172 of 200 right, or 86 percent. The Wilson interval, the method the NIST handbook describes for a proportion, gives about 80.5 to 90.1 percent at 95 percent confidence. Quote 86 percent without the 200 and you’ve hidden a spread of almost ten points.

Two candidates answered the same questions, so use that. In the same synthetic example, candidate B gets 161 of 200, or 80.5 percent, with an interval of about 74.5 to 85.4. The intervals overlap, and a reader might call it a tie. Pairing says more. McNemar’s test uses only the questions where the candidates disagree. With A alone right on 18 and B alone on 7, the statistic is (18 minus 7) squared, divided by 25, or 4.84, above the 3.84 needed for significance at the 5 percent level. An exact binomial version gives a p-value of about 0.043. Read those disagreements by hand too, whichever way the test comes out.

Your references are a measured quantity too

Published benchmarks help with orientation, and they’re less settled than their leaderboards suggest. A 2026 expert analysis reported annotation error rates of 52.8 percent in BIRD Mini-Dev and 62.8 percent in Spider 2.0-Snow. When 16 agents were rerun on a corrected BIRD Dev subset, their relative performance moved by between minus 7 and plus 31 percent, and ranks shifted by up to nine places.

Your internal set has the same disease on a smaller scale. When a candidate disagrees with a reference, audit the reference before recording a failure, and track how often the reference turns out wrong. That rate says how far to trust the score. The companion post on building an evaluation set covers how to source and own those references.

Pick a framework last

Frameworks run cases and collect scores. None of them knows what your revenue means, so I’d spend far more time on the references than on this choice. Decide by where evaluation will live in your process, and expect to write custom scorers for result-set comparison either way.

• RagasStrongest where retrieval is the concern. Its catalog covers context precision and recall, faithfulness and response relevancy, plus SQL metrics, and it allows modified or custom metrics.
• InspectAn open-source framework from the UK AI Security Institute and Meridian Labs, built around datasets, solvers and scorers. A solver can be a full tool-using agent, which suits multi-step analytics flows.
• promptfooAn open-source CLI and library with declarative test cases that runs locally and in CI. It suits side-by-side comparison across providers and adds red-team testing.
• Your own harnessA few hundred lines is a reasonable start when the grade is deterministic. Log the configuration with every run and you already have most of the value.

Turn the metrics into a scorecard this week

Write the scorecard before you see any results, because thresholds chosen afterwards tend to flatter the candidate someone already likes. Separate what blocks a release from what only gets reported, and keep the blocking list short. Set each number with the person who owns the decision. The values below are placeholders to show the shape, so replace them.

run:
  question_set: finance-pipeline@v14
  source_snapshot: 2026-09-30
  model: <exact pinned identifier>
gates:                     # any failure blocks the release
  permission_failures: 0
  result_correctness_min: 0.85
  false_answer_rate_max: 0.05
reported:                  # visible, never offsets a gate
  over_refusal_rate: 0.05
  numeric_grounding: 0.99
  table_recall: 0.92
  p95_latency_seconds: 18
  cost_per_accepted_answer_usd: 0.04
report_each_with: [n, wilson_95_interval, failed_case_ids]

How we report this

We run questions with known answers on each Nexatron release, and those runs can block it. We grade answers on returned results and score refusal in both directions. We publish no headline figure, because the figure moves with the model, and release runs pin the model production uses for the same reason.

Ask any system you evaluate for the question count, the reference process and the failed cases. A lone percentage with none of those attached isn’t evidence.