Abed ManzaljiAI, Step by Step

What's a reasoning model?

An AI that thinks before it answers. Better on hard problems, but slower and pricier.

4 min to watch2 min to read

Based on real research: OpenAI's reported results on the 2024 AIME math exam, September 2024,1 and OpenAI's docs on how hidden thinking is billed.3 The problems, the chats and the screens are made up.

Read the walkthrough as text

Without thinking first

  1. You're practicing for a math contest. You ask an AI chat for help.
  2. You type in a hard problem.
  3. This AI starts writing its answer straight away, in one go.
  4. Behind the scenes, there are no notes. It never stopped to plan or check.
  5. You look up the answer key. It says 47, not 53.
  6. The mistake: 15, 30, 45, 60, 75 and 90 divide by both 3 and 5. The AI counted them twice.
  7. It's not just this problem. OpenAI gave a real math contest, AIME 2024, to GPT-4o, an AI that answers in one go.
  8. With one try per problem, GPT-4o solved 9.3%. That's fewer than 1 in 10.
  9. Fast and confident isn't the same as right.

Fast isn't the same as right. The AI answered in one go and never checked. On 2024 AIME contest math, OpenAI reported that GPT-4o solved 9.3% of problems on its first try.

With thinking first

  1. Same problem, different AI. This one works the problem out before it answers.
  2. Behind the scenes, it writes notes to itself that you never see: a plan, then a first try.
  3. Then it checks its own work, and finds a problem.
  4. It fixes the mistake: take away the 6 numbers it counted twice.
  5. Then it tests its method on a small case it can count by hand.
  6. Only then does it answer. Slower, but right.
  7. This is a reasoning model: an AI that plans, tries, checks and fixes in hidden notes before it answers.
  8. On the real contest, OpenAI's reasoning model o1 solved 74.4%, 8 times as many. But it's slower, and its hidden notes are billed.

Think first on hard problems. A reasoning model plans, tries, checks and fixes in hidden notes before it answers. On 2024 AIME math, OpenAI's o1 solved 74.4% on its first try. The notes take time, and they're billed.

Thinking only when it pays

  1. Next, an easy question. You send it to the thinking AI too.
  2. Even for this, it writes hidden notes first. Fewer than before, but you still wait for them.
  3. The answer is right. But you paid for those notes too, and a fast AI gets this right at once.
  4. The fix: a quick sorting step looks at each question before any AI answers it.
  5. Easy, clear questions go to the fast AI. "12" comes back right away.
  6. Hard, many-step problems go to the thinking AI. It takes its time and gets 47.
  7. This is routing: a fast AI for easy questions, a thinking AI for hard ones. OpenAI says most AI workflows use both.

Think when it pays. Send hard, many-step problems to a reasoning model, and easy, clear ones to a fast model. You get right answers without paying for thinking you don't need.

Keep going:

Why it matters, what it costs, and what to ask your team.

The evidence

Same exam. Think first, score higher.

A regular chat model answers in one pass. Reasoning models are built to "spend more time thinking before they respond."2

That thinking happens in hidden notes. The model breaks a problem into steps, tries an approach, notices its mistakes and tries another.1

OpenAI tested this on AIME, a math exam built to challenge the best US high-school students.1 Switch between the tests:

9.3%GPT-4o, first try
74.4%o1, first try
8×as many problems solved

Reported by OpenAI on September 12, 2024. o1 was tested at its maximum thinking setting; o1-preview was the version released that day.

With 1,000 tries and a scoring model picking the best, o1 reached 93% on the same exam.1

It isn't better at everything. OpenAI's testers did not prefer o1-preview on some language tasks, "suggesting that it is not well-suited for all use cases."1

8 times as many solved. But how much slower, and how much pricier?

Try it

Which tasks need thinking?

Thinking costs time and money on every request. You even pay for the hidden working: it's billed like normal output.3

So sort these jobs. Fast model, or think first?

Our picks follow OpenAI's advice on when to use reasoning models.5 Your own tests should have the last word.

OpenAI's own advice: faster models for well-defined tasks where speed and cost matter, reasoning models for ambiguous, multi-step problems. Most AI workflows use both.5

So I think hard when the job is hard, and answer fast when it isn't.

Ask your team

Three questions before you pay for thinking.

Before switching everything to the smartest model, ask:

  1. Which of our tasks are actually hard?Multi-step, ambiguous or high-stakes work is where thinking pays. Route only those.
  2. Did we test it on our own hard cases?Benchmarks are a hint. Run both kinds of model on your real problems and compare.
  3. What's our limit on cost and wait?Hidden thinking is billed even though nobody reads it. Set a cap per request.3

Under the hood, step by step: build an app that sends each question to the right AI.

Build along

Build an app that picks the right AI, one step at a time

We start with one fast AI and the same three test questions, then improve it in small steps. Press Run it after each step and watch the result change.

The dial

More thinking, more waiting, a bigger bill.

Higher effort lets the model think more completely. It also uses more tokens and takes longer.3 Turn the dial:

Thinking effort
Answer you seeHidden thinking you pay for
0tokens per answer
0 swait per answer
$0per 1,000 answers

Made-up numbers to show the idea: a 300-token answer, 100 tokens a second, $10 per million output tokens. Real speeds, prices and thinking lengths vary by model and task.

  1. Measure on your own hard casesRun them at each effort level. Keep the lowest one that passes.
  2. Cap the outputThinking counts toward the output limit, so leave room for the answer itself.4
  3. Show progressA long silent wait feels broken. Stream the answer or show a "thinking…" state.
  4. Log thinking tokensTrack them per request so a cost spike gets noticed early.
  5. Re-check your routesNew models arrive often. Re-run the comparison when you switch.

References

  1. "Learning to reason with LLMs" · OpenAI, Sep 12, 2024. AIME 2024, GPQA Diamond and Codeforces results (Appendix A); human preference results.
  2. "Introducing OpenAI o1-preview" · OpenAI, Sep 12, 2024. Models "designed to spend more time thinking before they respond."
  3. "Reasoning models" · OpenAI API docs. Reasoning tokens are hidden but billed as output tokens; effort levels trade speed for thoroughness.
  4. "Extended thinking" · Anthropic, Claude API docs. A token budget for thinking, which counts toward the output limit.
  5. "Reasoning best practices" · OpenAI API docs. Faster models for well-defined tasks, reasoning models for complex ones; most workflows use both.

The problems, the answers, the chats, the screens, the task picks, the build-along's run output (times, tokens and costs) and the effort dial's numbers are made up to show the idea. Benchmark scores are as reported by OpenAI.

Made by Abed Manzalji, AI Engineer in Toronto.More about meContact