When AI makes something up and says it with total confidence.
4 min to watch1 min to read
Based on a real case: Mata v. Avianca, New York, 2023.1,2 The six fake case names are the ones from the real court filing. The screens, the chat lines and the two cases in the fixed version are made up.
Read the walkthrough as text
Without a search
You're a lawyer. Your client got hurt on a flight, and you need past court cases that back him up.
You ask an AI chat for help.
The AI doesn't search anything. It writes whatever sounds most likely, one word at a time.
It answers with a case that sounds just right, plus five more.
You paste the cases into your brief for the court.
Six cases, each with a name and an airline. It looks like real legal work.
Something feels off, so you ask the AI.
The AI says yes.
It never checked. It just gave the answer that sounded most helpful.
You file the brief. The airline's lawyers look up the cases. Nothing comes up.
The court looks too. None of the six cases ever existed.
This is an AI hallucination: the AI made something up and said it with total confidence.
The judge fined the lawyers $5,000.
That's an AI hallucination.ChatGPT invented six court cases, then said they were real. None existed. Real case: Mata v. Avianca, New York, 2023. The lawyers were fined $5,000.
With a search first
Same question. This time, the AI has to search a court case database before it answers.
First, it searches the case database. It finds two real cases, and reads them.
It answers using only what it found, with a link to each case.
You ask for one more.
It searches again. Nothing else matches.
So it says so, instead of guessing.
No guess and no made-up case. Just the truth: nothing else was found.
Your brief cites two cases. You open both links and check them yourself.
Look it up. Cite it. Or say "I couldn't find it."Making an AI search real documents before it answers is called RAG. It answers only from what it found, links every source, and says so when it finds nothing.
Keep going:
Why it matters, what it costs, and what to ask your team.
Why it happens
AI writes one likely word at a time.
A chat AI doesn't look facts up by default. It writes the next word that sounds most likely, then the next. It never stops to check if the whole sentence is true. Try it yourself:
The case that proves my client is right is
Word chances are made up to show how it works.
Every single word was a good guess. Together they make a court case that never existed.
In my defense, every word sounded right.
The real case, in numbers
Mata v. Avianca, 2023
This is exactly what happened to two New York lawyers.
0made-up cases filed in court
0confident "yes, they're real"
$0fine for the lawyers
The fix
Make it look things up first.
The fix has a name: RAG (retrieval-augmented generation). Before answering, the AI searches real documents and answers only from what it finds, with a link for every claim. Switch between the two:
If the search finds nothing, a good AI says so instead of guessing.
Under the hood, step by step: build a case finder that looks things up instead of making them up.
Build along
Build a case finder that can't make cases up, one step at a time
We start with the simplest version and ask it the same two questions every time. Press Run it after each step and watch the answers change.
So with RAG it never makes things up?
Much less often. I can still misread a source, so keep checking me.
Choosing what goes in
Too few results miss facts. Too many add noise.
Search returns chunks ranked by how close they are to the question. Only the top few go into the prompt. Slide to choose how many.
Question: "Do you have bereavement fares?" Scores are made up to show the idea.
Before you ship it
Small habits that catch the rest.
Keep chunks smallA few paragraphs each, with the document title attached.
The "with a search first" chapter, its two cases and the build-along's outputs are made up to show the fix. Charts marked as illustrations use made-up numbers.