Abed ManzaljiAI, Step by Step

What's AI bias?

When an AI learns an unfair pattern from old data, and treats some people worse because of it.

4 min to watch3 min to read

Based on a real case: Amazon's experimental résumé-rating tool, built from 2014 and reported by Reuters in 2018.1,2,3 The screens, candidates, résumés and star scores are made up.

Read the walkthrough as text

Without checks

  1. 2014. A team at Amazon starts building an AI to rate job applicants' résumés, to speed up the search for top talent.
  2. The dream, in one team member's words: give it 100 résumés, it picks the top five, "and we'll hire those."
  3. It gives each applicant one to five stars, much like shoppers rate products.
  4. To learn what "good" looks like, it studies résumés sent to the company over 10 years. Most came from men.
  5. It favors verbs more common on male engineers' résumés, like "executed" and "captured".
  6. It also learns a rule nobody asked for: the word "women's" means fewer stars.
  7. A new résumé comes in. Candidate B has the skills for the job.
  8. Then the AI reads one more line: captain of the women's chess club.
  9. Candidate B drops from four stars to two, to the bottom of the list. Same skills, one word.
  10. It also marks down graduates of two all-women's colleges.
  11. By 2015 the company sees the scores aren't gender-neutral. It edits the AI to ignore those words, but can't be sure that's enough.
  12. By early 2017 the team is disbanded. Amazon later says recruiters never used the tool to evaluate candidates.
  13. This is AI bias: an AI learns an unfair pattern from old data, and repeats it on new people.

That's AI bias. An AI learned from 10 years of résumés, mostly from men, and marked down the word "women's". Real case: Amazon's experimental résumé tool, reported in 2018.

With checks

  1. Same AI, same old résumés. This time it must pass fairness checks before it rates a single real applicant.
  2. Check 1: a paired test. Two copies of a résumé, the same except for one word.
  3. The second copy says "women's chess club". Nothing else changes.
  4. They don't match. The test fails, so the problem is caught before launch, not after.
  5. Check 2: look at which clues the AI relies on most. Words that point to gender get hidden, and it's retrained.
  6. Check 3: compare who gets shortlisted, by group, on test résumés. The rates must be close.
  7. Check 4: the AI only suggests. A person reviews each applicant and makes the call, and every decision is logged.
  8. The tests stay. They run again every time the AI is updated, so old habits can't sneak back in.

Test for bias before it meets real people. An AI copies the patterns in its examples, fair or not. Test pairs that differ by one word, compare results by group, keep a person deciding, and retest after every update.

Keep going:

Why it matters, what it costs, and what to ask your team.

Why it matters

A hiring AI that taught itself to prefer men.

From 2014, a team at Amazon built an experimental AI to rate job applicants from one to five stars, "much like shoppers rate products".1 It learned from résumés sent to the company over 10 years. Most came from men.1,3

0years of old résumés it learned from, mostly from men
0all-women's colleges whose graduates it marked down
55%of US HR managers expected AI in their regular work within 5 years (2017)

It marked down the word "women's", as in "women's chess club captain", and graduates of two all-women's colleges.1 Amazon edited it to ignore those words, but couldn't be sure it wouldn't find other unfair ways to sort people. The team was disbanded by early 2017.1

Amazon said recruiters never used the tool to evaluate candidates. People familiar with it said recruiters looked at its picks, but never relied on them alone.1 Either way, hiring is a big target for AI: in a 2017 survey, 55% of US HR managers said AI would be a regular part of their work within five years.1,2

Nobody told me to prefer men. I just copied the old pile.

Why it works

The AI copies the past.

An AI like this learns by example. Show it old résumés, and it learns to favor whatever they had in common, fair or not. As the ACLU put it, if software looks for résumés like the ones it learned from, "reproducing the demographics of the existing workforce is virtually guaranteed".4

What it learned from: mostly men 10 years of résumés A new applicant "women's chess club" AI one word changed Test ✗ caught, fixed 2 ★✗ marked down 4 ★✓ same as its pair

Hiding one word isn't enough. Other clues can point the same way: this tool also favored verbs more common on male engineers' résumés, like "executed" and "captured".1 So check who actually gets picked. Flip between three versions of the AI:

Made-up numbers to show the idea: 200 test résumés, 100 from women and 100 from men, top 40 shortlisted. The four-fifths line comes from US hiring guidelines.5

The ACLU's verdict on tools like this: they "are not eliminating human bias — they are merely laundering it through software."4 Checks are how you see it.

So the AI isn't neutral. It's a mirror of our old choices.

Ask your team

Four questions before an AI helps you hire.

None of this needs a big project. Start with these:

  1. What did it learn from?If the old decisions were unfair, the AI learns that too. This tool learned from 10 years of résumés, mostly from men.1
  2. Does one word change the score?Test pairs of résumés that differ only in a word like "women's". The stars should not move.
  3. Are results fair by group, before launch and after?Compare who gets picked. US guidelines flag a group picked at under four-fifths of the top group's rate.5
  4. Who makes the final call?The AI can sort the pile. A person should decide, and every decision should be logged.

Under the hood, step by step: build a résumé scorer that's tested for bias before it meets a real applicant.

Build along

Build a résumé scorer that's tested for bias, one step at a time

We start with the simplest scorer, run it on the same test résumés, and fix it in small steps. Press Run it after each step and watch the result change.

Before you ship

Five habits for any AI that scores people.

Whatever model you use, these keep an unfair pattern from reaching a real applicant:

  1. Write down what it must ignoreGender, and the words that point to it, like "women's" or a college's name.
  2. Test pairs, not just accuracySame résumé, one word changed: the stars must match.
  3. Check results by groupBefore launch on test résumés, and after launch on real outcomes.
  4. Keep a person decidingThe AI suggests. A person decides. Log both.
  5. Retest on every updateNew data can bring old habits back. The tests catch them.

References

  1. "Amazon scraps secret AI recruiting tool that showed bias against women" · Jeffrey Dastin, Reuters, October 10, 2018.
  2. "Amazon scrapped 'sexist AI' tool" · BBC News, October 10, 2018.
  3. "Amazon ditched AI recruitment software because it was biased against women" · Erin Winick, MIT Technology Review, October 10, 2018.
  4. "Why Amazon’s Automated Hiring Tool Discriminated Against Women" · Rachel Goodman, ACLU, October 12, 2018.
  5. "29 CFR 1607.4: Information on impact" · US Uniform Guidelines on Employee Selection Procedures, eCFR. The "four-fifths rule".

The screens, candidates, résumés and star scores are made up, and the order of events is simplified. The facts and the quotes come from the reports above. The "with checks" replay is made up to show how it should go: it is not what Amazon did. The shortlist rates in the chart and in the replay, and the build-along's code and outputs, are made-up numbers to show the idea.

Made by Abed Manzalji, AI Engineer in Toronto.More about meContact