A test your AI takes before it talks to real customers.
4 min to watch1 min to read
Based on a real case: Air Canada's help chatbot, 2024.1,2 The tribunal's words and the C$812.02 are real. The screens, the chat lines, the test and its scores are made up.
Read the walkthrough as text
No test caught it
An airline adds an AI help chat to its website, to answer customer questions.
A man's grandmother has just died. He needs a flight to the funeral, and asks about a discount.
The chatbot isn't sure of the airline's rule. So it writes what sounds helpful.
It answers kindly, and it sounds sure.
But that rule doesn't exist. The discount can't be claimed after the trip.
Its answer even links to the airline's own policy page. That page says the opposite.
He trusts the chat, buys full-price tickets and flies to the funeral.
After the trip, he asks for the discount. The airline says no, so he takes it to a tribunal.
The airline argues the chatbot is responsible for its own words, not the airline.
The tribunal disagrees. The chatbot is part of the airline's website, so the airline answers for it.
The airline has to pay him C$812.02. One wrong answer cost money, and made the news.
This really happened to Air Canada.Its chatbot gave a grieving customer a discount rule that didn't exist. In February 2024, a tribunal said the airline is responsible for what its chatbot says, and ordered it to pay C$812.02.
With a test
Rewind. Before launch, the chatbot answers 100 real customer questions. Each answer is checked against the airline's real rules.
One answer fails. It promises the funeral discount after the trip. The real rule says before.
This is an eval: a test of real questions the AI must pass before it meets customers. It scores 99 out of 100.
Not for a rule about money. The team puts the real rule in the chatbot's instructions and runs the same test again.
The scoreboard turns green. Now it's ready for customers.
Launch day. The same grieving customer asks the same question.
This time, the answer follows the real rule.
He knows the rule before he books. No customer ever sees the wrong one.
That test is called an eval.Real customer questions, checked answers and a score, before any customer sees a mistake. Run it again after every change.
Bonus: pick the AI
The same kind of test can also pick which AI to use. Here are three, from big and pricey to small and cheap.
Refund questions need 98 right out of 100. Only the big AI passes, so it gets that job.
Next job: sorting customer emails. That one only needs 90 out of 100.
All three pass. The small AI costs 40 times less, so it gets this job.
Same test. Different job. Different winner.An eval shows you the cheapest AI that's good enough for each job. These scores and prices are made up to show the idea.
Keep going:
Why it matters, what it costs, and what to ask your team.
How it works
An eval is a loop, not a one-time check.
You collect real questions, let the AI answer them, grade every answer, and fix what failed. Then you run it again whenever anything changes.
Tap a step.
Wait, I get tested every single time?
Why a quick try isn't enough
Small error rates hide at small scale.
Five test chats can all look perfect. At real volume, a small error rate turns into a lot of wrong answers. Drag the slider.
Chats a day
0
wrong answers every day
Each dot is 1% of chats. A simple multiplication, not real data.
That's why evals use hundreds of questions, not five.
300 wrong answers a day? That's 300 angry customers.
Picking the right AI
Same test. Different job. Different winner.
Evals also tell you which AI to use. Run the same test on a big, expensive model and a small, cheap one, then pick the cheapest one that's good enough.
The job
Made-up scores and prices to show the idea. The black line is the score the job needs.
Under the hood, step by step: build a test that catches a help chat's wrong answers before customers do.
Build along
Build a test for a help chat, one step at a time
The help chat stays the same in every step, with one wrong rule hiding in it. We only change the test. Press Run it after each step and watch what it catches.
Bigger tests
With hundreds of questions, set a pass line.
Our small test blocks a release if any question fails. Big tests often use a pass line instead. Drag it and see which releases would be stopped.
Made-up scores to show the idea.
Now a bad change can't sneak out on a Friday.
Grow it
Keep feeding it.
An eval gets better every week if you treat it like a product.
Start with 30 real questionsPull them from logs or support tickets, not from your imagination.
Every bug becomes a testWhen a customer finds a bad answer, add it so it can't come back.
Score by categoryRefunds, bags, pets. One average can hide a category that's failing.
Compare before you switchRun a new model or prompt through the same eval before it goes live.
The test, its questions, scores and prices, the "pick the AI" chapter and the build-along's outputs are made up to show the idea. Charts marked as illustrations use made-up numbers.