Build an evaluation set from decisions people make

Start from decisions people already make

Choose a bounded starting area such as sales pipeline, receivables, or regional performance. Ask users what they need to know and what they do with the answer. Preserve natural wording, including common ambiguity. The evaluation should reflect realistic requests while keeping the expected behavior explicit enough to grade consistently.

Build several kinds of cases

Include ordinary questions, edge cases, ambiguous requests, unsupported requests, and permission-sensitive cases. A complete answer is not the correct outcome for every prompt. Some cases should request clarification or decline within a clear boundary. Write the expected behavior before running the candidate system.

For each answerable case, record the measure, population, period, calculation, and acceptable tolerance. Have the source owner verify the reference. An incorrect reference can make a correct system appear wrong or reward a mistaken calculation. Treat reference quality as part of the evaluation process rather than assuming that any existing spreadsheet is authoritative.

Preserve important distinctions

Separate factual agreement, coverage, caveats, response structure, and business usefulness. A candidate may improve one dimension while regressing another. Report those differences instead of hiding them in a single average. Mandatory failures, such as unauthorized disclosure, should not be offset by strong performance on unrelated easy cases.

Include variations that change one condition at a time. Move a period boundary, alter a grouping, or switch an approved scope. These cases show whether the application preserves the rest of the question's meaning. They also make failures easier to diagnose because the reviewer can identify the specific condition that changed.

Protect the evaluation from overfitting

Keep a stable reference set for regression checks and a separate process for adding new cases. Avoid repeatedly tuning to a small public list until the results look perfect. The business requirement is broader than memorizing a handful of demonstrations. Use additional held-out or newly reviewed questions when assessing a meaningful change.

Control access to sensitive evaluation data. Synthetic cases can help test structure and boundary behavior, while approved real cases may be needed to represent the domain accurately. Label the evidence type clearly. A sample-data result should not be presented as proof of production performance or a customer outcome.

Keep ownership after the pilot

Assign someone to review new failure reports and decide whether they belong in the set. Update references when legitimate business definitions change, preserving the reason. A stale evaluation can block a correct improvement or allow a regression because it no longer represents the workflow people use.

For Nexatron, bring a small set of business-owned questions to the first evaluation and expand it as evidence accumulates. Aim for a repeatable way to decide whether a configuration supports the intended decisions, with enough detail to see what improved, what failed, and what to do next. A single accuracy number says little on its own.

Include a known disagreement

Add one question for which two teams currently use different legitimate definitions. The expected behavior may be to ask for clarification or use a named approved convention. This case tests whether the application helps resolve a real organizational ambiguity instead of merely reproducing a spreadsheet. It also gives evaluators a reason to improve the business reference itself when disagreement appears. Keep the resolution and its owner attached to the case so later reviewers understand why it changed.

Keep one ordinary case unchanged while adding the disagreement. This helps reviewers distinguish improvement in clarification behavior from an accidental regression in a previously supported calculation.