All articles
ai-evaluationdata-leadershipanalyticsdata-teamsai-adoption

Your Senior Analysts Are Your Best AI Evaluators

An AI answer can be fluent and still wrong for your business. The people who catch it are your senior analysts. Turn their hardest questions into a test set and measure the tool against it.

John SteinmetzNovember 20, 20264 min read
Your Senior Analysts Are Your Best AI Evaluators

An AI tool that answers in plain English will sometimes be wrong in plain English. Your senior analysts can tell when that happens. Make their judgment the test set, and you can measure the tool instead of hoping it works.

Fluent is not the same as right

A wrong AI answer does not look wrong. It has clean sentences, a number, and a reason. It reads like the work of someone who knows the business.

Take a question like "How many customers renewed last quarter?" The tool finds a table, counts the rows, and explains the result in two tidy lines.

A senior analyst reads the same answer and asks one thing first. Do customers who changed plans count as renewals? The table does not say. The analyst knows that the answer changes the number.

That check takes the analyst a few seconds. The tool did not ask it at all.

Demo questions test the demo

Most AI tools get judged on the questions in the demo. Someone picked those questions because the tool answers them well.

That tells you the tool can work. It does not tell you where it fails. Your real users will ask the odd questions, the ones with history behind them.

The people who know those questions are the analysts who have answered them for years.

Build the test set from their memory

A test set is a fixed list of questions with known good answers. You run the tool against it and count how many it gets right.

Ask each senior analyst for the questions that trip people up:

  1. Which question do new hires get wrong?
  2. Which number has a catch that is not in the table?
  3. Which question has two correct answers, depending on the team that asks?
  4. When is the honest answer "the data cannot tell you that"?

Keep the first list small. Each analyst should be able to review every answer by hand.

Write the answer key, not only the answer

For each question, the analyst writes the right answer and the reason for it. The reason is the part a fluent wrong answer leaves out.

Then grade each tool answer on the same short scale:

  • Right.
  • Wrong.
  • Right number, wrong reason.
  • Should have said it does not know.

The third grade is the dangerous one. It passes a quick look, and it fails the next time the data changes.

In October, I wrote that you should keep the analysts who already know the business. This answer key is the job that makes that case concrete.

Run it every time something changes

A test set is only useful if you run it again. Run it when the model changes, when a prompt changes, and when a source table changes.

Track the score over time. If it drops, you know before a leader finds the problem in a meeting.

When the tool gets a real question wrong, add that question to the set. The test set grows from real mistakes, not from guesses.

The key also depends on clear definitions. If nobody owns the metric, your analysts cannot write one right answer for it. Name the owner first.

The analysts become the standard

This changes how a team sees its senior analysts. They are not the slow path that AI was supposed to replace. They are the standard the AI has to meet.

It also gives you an honest answer when a leader asks if the AI is ready. You do not say that it looks good. You show the score on questions your best people wrote.

Your best AI evaluator already works for you. It is the person who asks whether a plan change counts as a renewal.

John Steinmetz logo
JOHN STEINMETZ
Fractional CDO

Fractional CDO services delivering data architecture, governance, and practical AI implementation that drives measurable enterprise value.

Navigate
Schedule a MeetingAbout
Connect
LinkedIn
Clients
Portal Login
© 2026 John Steinmetz / Fractional CDO
Message ConsentAdminArchitecting Intelligence