An AI tool that answers in plain English will sometimes be wrong in plain English. Your senior analysts can tell when that happens. Make their judgment the test set, and you can measure the tool instead of hoping it works.
Fluent is not the same as right
A wrong AI answer does not look wrong. It has clean sentences, a number, and a reason. It reads like the work of someone who knows the business.
Take a question like "How many customers renewed last quarter?" The tool finds a table, counts the rows, and explains the result in two tidy lines.
A senior analyst reads the same answer and asks one thing first. Do customers who changed plans count as renewals? The table does not say. The analyst knows that the answer changes the number.
That check takes the analyst a few seconds. The tool did not ask it at all.
Demo questions test the demo
Most AI tools get judged on the questions in the demo. Someone picked those questions because the tool answers them well.
That tells you the tool can work. It does not tell you where it fails. Your real users will ask the odd questions, the ones with history behind them.
The people who know those questions are the analysts who have answered them for years.
Build the test set from their memory
A test set is a fixed list of questions with known good answers. You run the tool against it and count how many it gets right.
Ask each senior analyst for the questions that trip people up:
- Which question do new hires get wrong?
- Which number has a catch that is not in the table?
- Which question has two correct answers, depending on the team that asks?
- When is the honest answer "the data cannot tell you that"?
Keep the first list small. Each analyst should be able to review every answer by hand.
Write the answer key, not only the answer
For each question, the analyst writes the right answer and the reason for it. The reason is the part a fluent wrong answer leaves out.
Then grade each tool answer on the same short scale:
- Right.
- Wrong.
- Right number, wrong reason.
- Should have said it does not know.
The third grade is the dangerous one. It passes a quick look, and it fails the next time the data changes.
In October, I wrote that you should keep the analysts who already know the business. This answer key is the job that makes that case concrete.
Run it every time something changes
A test set is only useful if you run it again. Run it when the model changes, when a prompt changes, and when a source table changes.
Track the score over time. If it drops, you know before a leader finds the problem in a meeting.
When the tool gets a real question wrong, add that question to the set. The test set grows from real mistakes, not from guesses.
The key also depends on clear definitions. If nobody owns the metric, your analysts cannot write one right answer for it. Name the owner first.
The analysts become the standard
This changes how a team sees its senior analysts. They are not the slow path that AI was supposed to replace. They are the standard the AI has to meet.
It also gives you an honest answer when a leader asks if the AI is ready. You do not say that it looks good. You show the score on questions your best people wrote.
Your best AI evaluator already works for you. It is the person who asks whether a plan change counts as a renewal.
