Abed ManzaljiAI, Step by Step

Does AI make developers faster?

Not always. Feeling faster and being faster are different things, so measure it.

4 min to watch2 min to read

Based on real research: METR's randomized trial with 16 experienced open-source developers and early-2025 AI tools,1,2 and METR's 2026 update on it.3 The developer, the bug, the team and the screens are made up.

Read the walkthrough as text

Going by feel

  1. You're an experienced developer. You've worked on this big project for years.
  2. Today you have an AI assistant in your code editor. You ask it to fix a bug.
  3. Before starting, you guessed this would take about an hour without AI. The clock starts at 9:00.
  4. The AI writes 48 lines of code in a few seconds. It feels amazing.
  5. You paste it into the editor and run the tests. Three fail: December comes out as January.
  6. You explain the project's own rule to the AI, then wait for a new version.
  7. It writes a new version just as fast. Still feels quick.
  8. Read it, test it, fix it, repeat. Each round feels short, so the whole job feels fast.
  9. Merged! It felt fast. But the clock says 71 minutes, longer than your 1-hour guess without AI.
  10. This really happened in a 2025 trial: 16 experienced developers, 246 real tasks, a coin flip for AI or no AI.
  11. Before, they expected AI to cut their time by 24%. Afterwards, they believed it had cut 20%.
  12. The clock said the opposite: with AI allowed, tasks took 19% longer.
  13. Typing and searching took less time. Prompting, waiting for the AI and checking its work took more.

Feeling faster isn't being faster. In METR's 2025 trial, experienced developers expected AI to cut their time by 24%, and afterwards believed it had cut 20%. Timed, tasks with AI allowed took 19% longer.

With a fair test

  1. Your team wants to know if AI really helps on its own work. So it runs a fair test.
  2. First, each task gets a time guess, made before anyone knows how it will be done.
  3. Then a coin flip decides each task: AI allowed, or no AI.
  4. The coin keeps flipping, so easy and hard tasks end up in both groups.
  5. Everyone works as usual. A month later, 200 tasks have been timed from start to merged.
  6. You compare each group's real time with its guesses. On new tests and small scripts, AI helped.
  7. On old core code the team knows by heart, AI slowed them down, much like the 2025 trial found.
  8. Now the numbers decide where to use AI, not the feeling. The coin keeps flipping, because tools keep changing.

Don't trust the feeling. Time it. Guess first, flip a coin per task, time every task to merged, then compare. In early 2026, METR said developers are likely sped up more than in early 2025, so measure your own work.

Keep going:

Why it matters, what it costs, and what to ask your team.

The surprise

Feeling faster isn't being faster.

In 2025, the research group METR ran a randomized trial. 16 experienced developers did 246 real tasks on big open-source projects they had worked on for about 5 years.1,2

Before each task, a coin flip decided: AI allowed, or not. Then the time was measured. Experts forecast the result too. See how far off everyone was:

← more timeless time →

Forecasts, beliefs and the measured result from METR's 2025 trial. The whisker shows the likely range for the measured result.2,3

They mostly used Cursor with Claude 3.5 and 3.7 Sonnet, top tools in early 2025.2 Even after the trial, the developers believed AI had saved them time.

In my defense, I felt fast too.

Why it happens

The typing got faster. The job didn't.

With AI allowed, developers spent less time writing code and searching. They spent more time prompting, waiting for the AI and reviewing what it wrote.2 Switch between the two:

Read code Write it Test Done slow typing, but one pass Prompt Wait Review Test Fix it Done repeat until it's right typing is fast; the loop is not

They accepted under 44% of what the AI wrote, and about 9% of their time went to reviewing and cleaning its output. The slowdown was bigger on tasks they were more familiar with.2

So the minutes didn't vanish. They moved from typing to checking.

Ask your team

Three questions before you buy seats for everyone.

It's not just one study. In Google's 2024 DORA research, more AI use went with faster code reviews and better docs, but slightly lower delivery throughput and stability.4

Source: Google's DORA research, 2024. The gains are for a 25% rise in AI adoption.

  1. How do we know it's faster?In the trial, developers felt faster while being slower. Ask for timed tasks, not a show of hands.
  2. Faster at which part?Writing code is one step. Ask about review, testing, fixes and shipping, from start to finish.
  3. Who checks the AI's code?More code, faster, means more to review. Ask how review time and bugs have changed.

Under the hood, step by step: build a fair test of AI on your own team's work.

Build along

Build a fair test of AI on your team, one step at a time

We start with the usual show of hands and turn it into a fair trial in small steps, using the same month of team tasks. Press Run it after each step and watch the result change.

Before you trust it

Enough tasks, and a real review.

How many tasks do you need? More than you'd hope. Run this pretend trial a few times and watch the estimate jump.

A made-up simulation. Each dot is one task: its real time divided by its estimate. Tasks vary a lot, so small trials are noisy.

With 20 tasks the answer swings wildly. With a few hundred it settles. METR used 246 tasks, and its likely range was still 2% to 39% more time.3

And review AI-written code like a stranger wrote it:

  1. Keep changes smallSmall pull requests get real reviews. DORA's research points back to small batches and solid tests.4
  2. Label AI-assisted changesOne checkbox in the pull request is enough to compare groups later.
  3. Tests come with the codeIf the AI wrote it, a test a human has read shows what it really does.
  4. Don't skim tidy codeCheck edge cases, names and error handling. Neat-looking code can still be wrong.
  5. Watch the reworkCount review rounds and reverts per group. That's where hidden time lives.

References

  1. "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity" · METR, July 10, 2025. 16 developers, 246 tasks; 19% more time with AI; developers expected 24% less and believed 20% less.
  2. Becker, Rush, Barnes, Rein. "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity" · arXiv:2507.09089, July 2025. The full paper: expert forecasts, tools used, time spent, acceptance rate, likely causes.
  3. "We are Changing our Developer Productivity Experiment Design" · METR, Feb 24, 2026. The 2025 result's likely range (2% to 39% more time), and why a late-2025 follow-up gave an unreliable signal.
  4. "Highlights from the 10th DORA report" · Google Cloud blog, Oct 22, 2024. AI adoption, delivery throughput and stability.

The developer, the bug, the editor log, the team, the chats, the "fair test" results and the build-along's numbers are made up to show the method. The trial simulation is an illustration, not data. What this doesn't cover: the 2025 trial studied experienced developers on large projects they knew well, with early-2025 tools. METR's follow-up with late-2025 tools hinted at a speedup, but METR called that data unreliable because many developers no longer wanted to work without AI. METR believes developers are likely sped up more now than in early 2025. Your team, your code and today's tools may differ. That's why you measure.

Made by Abed Manzalji, AI Engineer in Toronto.More about meContact