What to ask an AI analytics vendor before you buy
6 min readUpdated
The questions that matter most in an AI analytics evaluation are about what happens when things go wrong: what the tool does when the data cannot answer, whether it shows where a number came from, whether permissions are enforced automatically per person, and what the first ninety days actually require from your team. A demo on the vendor's data proves very little; ask to see the same questions on yours.
Before the demo
- What does the first 90 days require from us?Ask for it in named roles and days. "Minimal setup" that turns out to need a data engineer for two months is the single most common source of stalled rollouts.
- Which of our systems can it read on day one?Name yours specifically. "We integrate with everything" usually means a connector exists for the popular half and the rest is a project.
- Does our data leave our environment?A yes is not automatically wrong, but your security team will ask, and the answer determines how long the review takes.
During the demo, on your data, not theirs
- Ask something you already know the answer toThe single most useful test. If the number is wrong you will know instantly, and if it is right you have learned something about the harder questions.
- Ask for something your data does not containWatch what happens. A good tool tells you exactly what it could not find. A dangerous one produces a confident, well-formatted answer that is entirely invented.
- Ask the same question two different waysIf the two answers disagree, the definitions are not settled and every number it produces is a coin flip you cannot see.
- Ask where a number came fromIf it cannot show you, your finance team will never present it, and the tool becomes a curiosity rather than infrastructure.
After the demo
- Can it be logged in as someone junior?Have them ask a question about executive compensation. Whether access is enforced automatically, per person, decides whether this can be rolled out beyond a pilot group.
- What does it cost when everyone uses it?Pricing that works for twenty people sometimes does not for two thousand. Ask for the shape of the curve, not just this year's number.
On accuracy claims
Most vendors will quote an accuracy figure. It is worth knowing that these are very hard to compare: the number depends on which question set was used, which AI model was configured, and what counted as correct, and the published industry benchmarks have documented labelling errors of their own.
You do not need to litigate this. Just ask three follow-ups: measured on which questions, using which model, and how often does it wrongly refuse a question it should have answered? A vendor who can answer all three has done the work. One who cannot has a number from a slide.