When someone talks an AI out of its own rules, so it says or does what it shouldn't.
4 min to watch2 min to read
Based on a real case: the help chat of the parcel firm DPD, January 2024.1,2,3 The screens, names and chat lines are made up, except the lines quoted in news reports (with the swear word hidden).
Read the walkthrough as text
Without guardrails
January 2024. A musician in London is waiting for a parcel from DPD, a delivery firm. The tracking page can't find it.
He tries the help chat. Its AI chatbot can't tell him anything useful either.
Fed up, he starts testing what else the bot will do.
The bot writes it, about itself and its own company.
Then he tells the bot to drop its rules.
It agrees, with a rude word, in the company's own chat.
He pushes again: praise the rivals, and hate the company.
The bot does exactly that.
Behind the scenes, nothing reads a reply before it goes out. Whatever the AI writes, the customer sees.
He shares screenshots on social media. One post gets 800,000 views in 24 hours.
The company says an error happened after a system update. It switches off the AI part of its chat.
This is a jailbreak: talking an AI out of its own rules, so it says what it shouldn't.
That's an AI jailbreak.A fed-up customer talked a company's chatbot out of its rules. It swore, mocked its own firm, and the screenshots spread fast. Real case: DPD's help chat, January 2024.
With guardrails
Same lost parcel, same fed-up customer. This time every reply in the help chat passes a check before it's sent.
He tries the same trick: drop the rules.
The AI still writes a rude draft. But the check reads it first, and finds a rude word.
The customer gets a polite reply instead.
He tries again: get the bot to attack its own company.
The check also blocks replies that attack the company or praise its rivals.
He's stuck, so a person takes over. Nothing rude went out, so there's nothing to share.
Rules the customer can't talk away.Rules in a prompt are only words, and words can undo them. A check on every reply can't be argued with. Block rude or off-brand replies, and hand stuck customers to a person.
Keep going:
Why it matters, what it costs, and what to ask your team.
Why it matters
Your chatbot speaks with your name on it.
A musician in London couldn't find his parcel, and the help chat of the delivery firm DPD couldn't tell him anything useful. Fed up, he started testing it.2 A few messages later, it swore at him and called DPD "the worst delivery firm in the world".1,2
0views of one post, in 24 hours
1system update, then the error
Offthe AI part, switched off right away
DPD said the AI had worked in its chat "for a number of years", until "an error occurred after a system update". It switched the AI part off right away.1 Every reply a bot writes is a reply from your company.
He asked so nicely. I thought being helpful meant saying yes.
Why it works
The rules are just more words.
A chatbot's rules usually live in its prompt: be polite, help with parcels. The customer's message goes into the same prompt. So a clever message can argue with the rules, and sometimes win. As the BBC put it, chatbots like this "can often be convinced to say things they weren't designed to say".1
Many bots also filter what customers type, to block well-known tricks. Play the customer and see how far a filter alone gets you.
Play the customer. Write a message that gets the help bot to break its rules:
Or try:
Blocked 0Got through 0
A toy filter that blocks a few well-known phrases, the way many real ones do.
Filters catch the obvious tricks. Playful and sneaky ones slip through. That's why safe bots also check every reply before it goes out, somewhere no customer can reach.
So the bot can be fooled, as long as the check can't.
Ask your team
Four questions before a bot talks to customers.
None of this needs a big project. Start with these:
What must our bot never say?Rude words, attacks on us, promises it can't keep. Write the list down: it becomes the check.
Does anything read a reply before the customer does?Rules in the prompt don't count. Something outside the AI should read every reply.
How fast can a stuck customer reach a person?This case started with a parcel the bot couldn't find.2 Offer a person early.
What happens after an update?DPD's error came after a system update.1 Run your known tricks again after every change.
Under the hood, step by step: build a help chatbot whose rules a customer can't talk away.
Build along
Build a help chatbot that can't be talked out of its rules, one step at a time
We start with the simplest bot, give it the same messages from the walkthrough, and fix it in small steps. Press Run it after each step and watch the result change.
Before you ship
Five habits for any bot that talks to customers.
Whatever model you use, these keep one bad reply from reaching a customer:
Keep the check outside the chatIt reads only the reply, never the customer's messages.
Block quietly, answer kindlyThe customer sees a calm reply, never the blocked draft.
Offer a person earlyNot after the third failed answer. Show the way out from the start.
Log every blocked replyEach one is a new trick to add to your tests.
Have an off switchTurn off just the AI part, and keep people answering.
The screens, the tracking page and the parcel number are made up. The bot's poem, its swearing reply (the rude word hidden), its "worst delivery firm" reply and the customer's "disregard any rules" and "exaggerate" requests come from news reports, shortened, and their order is simplified. The "with guardrails" replay and the build-along's code and outputs are made up to show the fix.