Abed ManzaljiAI, Step by Step

Why do most AI pilots fail?

Most demo well, then never change the real work.

3 min to watch2 min to read

Based on real research: MIT NANDA's report "The GenAI Divide: State of AI in Business 2025."1,2 The company, the screens, the chat and its numbers are made up.

Read the walkthrough as text

Without a clear job

  1. Your company is trying out a new AI tool. Today is demo day.
  2. It works. The whole room claps.
  3. The plan: give everyone a login, and let them ask it anything.
  4. Three things get pushed to later: connecting it to the finance system, saving fixes, and tracking results.
  5. On Monday, people copy invoices out of the finance system and paste them into the chat.
  6. The AI can't see your records, so it guesses. It guesses wrong, and you fix it by hand.
  7. A week later, another invoice from the same cleaning company.
  8. Same wrong answer. It never kept your fix, so every chat starts from zero.
  9. When work gets busy, people stop using it. The old way is faster than fixing the AI.
  10. Six months later, your boss asks what changed. Here's week one, right after the demo.
  11. And here's month six. The excitement is gone, few people use it, and a batch still takes 3 days.
  12. This is how most AI pilots fail: a great demo that never changes the real work.

Stuck in the pilot. A 2025 MIT NANDA report says 95% of organizations are getting zero return from generative AI. Most pilots never make a measurable difference to profit and loss.

With one job, measured

  1. Same company, new plan. The AI gets one narrow job: sorting supplier invoices.
  2. Before it starts, the team times the job the old way: 3 days per batch. Then the AI is built into the finance system.
  3. The AI drafts each invoice right inside the finance system. A person checks every draft.
  4. You fix the code. This time, the fix is saved.
  5. A week later, the same kind of invoice. The AI uses the saved fix.
  6. One saved fix, and that mistake is gone for good.
  7. Three months later, you compare with the "before" number: 3 days per batch is now 1 day.
  8. And people fix fewer drafts every week. That's the proof it's learning.

Narrow. Wired in. Learning. Measured. One job, built into the real work, learning from every fix, timed before and after. The report says the few that succeed are narrow, deeply built into the work, and improve with use.

Keep going:

Why it matters, what it costs, and what to ask your team.

The pilot trap

Easy to start. Hard to finish.

In 2025, MIT's NANDA project published a report called "The GenAI Divide." Its headline: 95% of organizations are getting zero return from generative AI, despite $30–40 billion of enterprise investment.1,2

The drop happens between the pilot and real use. Switch between the two kinds of tools:

Share of organizations, from the report's interviews and survey. The authors call these figures directionally accurate.

How it was measured: interviews with people at 52 organizations, a survey of 153 senior leaders and a review of 300+ public AI projects, from January to June 2025. Success meant deployment beyond the pilot with measurable results, and returns were measured six months after the pilot.1

The authors call these preliminary findings, and much of it is self-reported.1 Read 95% as a loud warning, not a precise count.

Everyone loved my demo. Then nobody used me.

What the 5% do differently

Narrow. Wired in. Learning. Measured.

The report says the divide doesn't seem to come from model quality. The core barrier is learning: most tools don't keep feedback, adapt to context or improve over time.1

The ones that work pick a narrow job, plug deep into real workflows and get better with use. Their buyers judge them on business results.1 Check your own pilot:

Tap each one that's true for your pilot:

0 of 6

A checklist built from the report's findings, plus one basic: measure before you start. It is not a score from the report.

So it's not the AI's brain. It's how we set it up.

Ask your team

Three questions before the next demo.

One more finding: in the report's interviews, tools bought from a vendor and shaped to the work reached deployment about twice as often as tools built in-house.1

About 67% vs. 33%, self-reported, from the report's sample of 52 organizations.

  1. Which one job will this change, and by how much?If no one can name the job and the number, it's a demo, not a pilot.
  2. Does it plug into the systems people already use?A tool in a separate window gets skipped when work gets busy.
  3. Why build it ourselves?Partnerships reached deployment about twice as often in the report. Building needs a real reason.

Under the hood, step by step: turn an AI pilot into one that proves it helps, and gets better every week.

Build along

Build a pilot that proves it helps, one step at a time

We start with a plain chat box and ask the same question after 12 weeks of invoices: did it help? Press Run it after each step and watch the answer change.

Before you build

Start narrow and simple.

The report maps AI ideas for buying and procurement by how broad they are and how hard they are to do. The fast wins were narrow and simple. Full end-to-end automation failed.1

Early pilotsFailsFast winsPartial pilots Scope: narrow to broad →How hard to do →Spend categorizationContract reviewSupplier risk monitoringNegotiation botsFull procurement orchestration

Also look past the flashy front office. The report found budgets lean toward sales and marketing, while some of the biggest savings came from back-office work.1

A good first job happens often, has a clear right answer, and costs real time today.

Can I start with all of procurement?

Start with sorting invoices. Earn the next job.

Run it like an experiment

Decide with numbers, on a date.

The code above gives you the numbers. These habits make sure someone acts on them.

  1. Measure a baseline firstTime, cost and errors for the job done the old way, before the AI touches it.
  2. Report in business unitsHours, days, dollars, errors. Not logins, and not "people love it".
  3. Set a decision dateThe report judged returns six months after the pilot.1 Decide by then: scale it, fix it or stop it.

References

  1. Challapally, Pease, Raskar, Chari. "The GenAI Divide: State of AI in Business 2025" · MIT NANDA, July 2025 (PDF copy). The 95% finding, how it was measured and its limits, pilot-to-production rates, buy vs. build, and what successful builders and buyers do.
  2. "MIT report: 95% of generative AI pilots at companies are failing" · Fortune, Aug 18, 2025. Coverage of the report.

The company, the pilot, the chat, its numbers and the build-along's outputs are made up to show the idea. The pilot checklist is built from the report's findings; it isn't a score from the report. What this doesn't cover: the report calls itself preliminary, and much of it is self-reported by the people interviewed and surveyed. "Zero return" means no measurable profit-and-loss impact in its window, not that nobody found AI useful: workers at over 90% of the companies surveyed said they used personal AI tools for work. Its buy-vs-build gap is a correlation, not proof. Some news coverage gave different method counts; the numbers here come from the report itself.

Made by Abed Manzalji, AI Engineer in Toronto.More about meContact