When a model keeps predicting yesterday's world, long after the world has changed.
4 min to watch3 min to read
Based on a real case: Zillow Offers, the home-buying business Zillow decided to wind down in November 2021.1,2,5 The screens, homes, prices and the model's lines are made up.
Read the walkthrough as text
What happened
2018. Zillow, the big home-listing website, starts buying homes for cash, to fix up and sell again.
Millions of homes on Zillow show a Zestimate: a value worked out by machine learning from records, home details and photos.
February 2021. For some homes in over 20 cities, the Zestimate becomes Zillow's first cash offer.
Behind the scenes, a model learns from past sales. In the pandemic, home prices rose at an unprecedented rate.
To buy a home, Zillow needs more than today's value: what will it sell for after repairs, three to six months later?
July to September 2021. Zillow buys 9,680 homes, more than twice as many as the three months before.
But home values are changing faster than ever before. The forecasts start to miss.
Zillow later said its forecasts missed, in both directions, by much more than it had modeled as possible.
More and more homes are now worth less than their forecasts said. The gap keeps growing.
Selling can't keep up: 3,032 homes sold in those three months. At the end of September, 9,790 are still waiting.
October 2021. Zillow stops signing new deals for the rest of the year, citing a backlog of homes to fix and sell.
November 2021. Zillow writes down about $304 million, decides to shut Zillow Offers down, and plans to cut about 25% of its staff.
Its CEO explains: "the unpredictability in forecasting home prices far exceeds what we anticipated."
This is model drift: the world changes, the model keeps predicting the old one, and the gap shows up as losses.
That's model drift.Zillow's forecasts said what homes would sell for months later. Home values changed faster than ever, and it bought homes for more than they'd sell for. Real case: Zillow Offers, 2021.
With monitoring
Same model, same fast-changing market. This time, every forecast is checked against what the home really sells for.
Each sale is logged next to its forecast. Small misses are normal. The limit is 5%.
Weeks later, the market changes. The misses grow, and the check notices: past the limit, the team gets an alert.
Buying slows down on its own: lower offers, fewer homes, until the forecasts are back on track.
Big bets go to a person. An offer over $500,000 waits until someone on the team says yes.
At 8% off, buying pauses. The model is retrained on the newest sales, then tested before it goes back to work.
Buying starts again, at prices that fit today's market. The gap stays small, and no pile of overpriced homes builds up.
Check the model against the real world.A model only knows the world it learned from. Compare every forecast with what really happened, slow down when the misses grow, let a person approve big bets, and retrain on fresh data.
Keep going:
Why it matters, what it costs, and what to ask your team.
Why it matters
A home-buying model that bet on the old market.
In February 2021, Zillow made its Zestimate, a machine-learning estimate of a home's value, the first cash offer on some homes in more than 20 cities.4,5 Its home-buying business, Zillow Offers, bought homes to fix up and sell again.5
0homes Zillow bought from July to September 2021
0million written down: homes it paid more for than they'd sell for
25%of its staff to be cut as it wound the business down
From July to September it bought 9,680 homes, and sold 3,032.2,7 It said it had bought homes in those months at higher prices than it now expected to sell them for, and wrote down about $304 million.1,3 Then it decided to wind Zillow Offers down, and planned to cut about 25% of its staff.1
The CEO's explanation: "the unpredictability in forecasting home prices far exceeds what we anticipated."1 Zillow also blamed a backlog of homes to fix and sell, made worse by labor and supply shortages.3,6 The model was one part of it. But every wrong forecast was a real home, bought at the wrong price.
I learned last year's market by heart. Then I kept pricing homes for it.
Why it works
The world moves. The model doesn't.
A model learns from examples, and every example is from the past. When the world changes, the model doesn't know. It keeps answering, just as sure as before. Zillow told its shareholders that its forecasts had missed, in both directions, by much more than it had modeled as possible.2
A sketch to show the idea, not real prices.
Zillow's words for its challenge: forecasting prices "three to six months out, in a market where there were larger and more rapid changes in home values than ever before."5 You can't stop the world from moving. But you can notice, fast, when your model falls behind. Drag the line to set when the alarm rings:
Made-up numbers to show the idea. Each bar is the share of homes that sold within 5% of their forecast that month.
Set it too low, and the alarm rings after the money is gone. Set it too high, and it rings every month, so people stop listening. Pick a line, and decide in advance what happens when it rings.
So a model is never finished. It needs a check-up against the real world, all the time.
Ask your team
Four questions before a model makes big bets.
None of this needs a big project. Start with these:
Do we compare every prediction with what really happened?Log each forecast next to the real result. Without that, the gap only shows up as a loss.
How big a miss is too big, and who hears about it?Pick the limit before launch. Decide who gets the alert, and how fast they must act.
What slows down on its own?When the misses grow, bet less automatically: lower offers, fewer of them, or a pause. A person approves the biggest ones.
When was it last trained, and on what?Markets move. Zillow said home values changed faster than ever before.5 Retrain on recent data, and test it before it goes back to work.
Under the hood, step by step: build a home-pricing model that notices when the market moves, before the losses do.
Build along
Build a home-pricing model that notices when the market moves, one step at a time
We start with a model that prices homes and buys them, run it on the same made-up months as the market changes, and add one small check per step. Press Run it after each step and watch the result change.
Before you ship
Five habits for any model that makes real bets.
Whatever model you use, these keep a stale model from spending real money:
Log predictions next to outcomesEvery forecast, and what really happened, in the same row.
Watch a rolling windowThe last 30 days, not all time. Old wins can hide new misses.
Give every limit an actionPast the limit: tell a person, then slow down or pause on its own.
Keep a person on the big betsAbove a set size, the model suggests and a person decides.
Retrain on fresh data, then testThe new model must pass on recent results it hasn't seen before it ships.
"Q3 2021 Shareholder Letter"· Zillow Group (SEC Form 8-K, Exhibit 99.3), November 2, 2021. Homes bought, sold and still in inventory, and the pause on new contracts.
The screens, homes, addresses, prices and the model's lines are made up, and the order of events is simplified. The home counts, the $304 million write-down, the 25% staff cut, the dates and the quotes come from the sources above. We don't know how Zillow checked its models inside: the "behind the scenes" lines only show the idea. The "with monitoring" replay is made up to show how it should go: it is not what Zillow did. The alarm chart, and the build-along's code and outputs, are made-up numbers to show the idea.