Abed ManzaljiAI, Step by Step

Prompt, RAG, or fine-tune?

Three ways to teach an AI your business: a better prompt, look-ups, or training.

4 min to watch2 min to read

Based on real research: BloombergGPT, 2023,1 and a 2023 study that gave GPT-4 the same finance tests.2 The company, its rates, the chats and the screens are made up.

Read the walkthrough as text

Training first

  1. You lead a team at a finance company. You want an AI that really knows finance, and your products.
  2. The AI likes the idea. It sounds like the most serious option.
  3. So the team trains a new model from scratch, on about half finance text and half general text. Bloomberg did this in 2023.
  4. Bloomberg's real run used 512 GPUs, powerful computer chips, for about 53 days.
  5. Weeks go by. The chips run day and night, and the bill keeps growing.
  6. Day 53: done. Bloomberg published its model's scores on five public finance tests.
  7. Here are three of those tests. BloombergGPT's published scores come in.
  8. Then researchers gave the same tests to GPT-4, a general AI with no finance training, using only prompts.
  9. GPT-4 beat BloombergGPT's published scores on 4 of the 5 tests. The test setups weren't always identical.
  10. And one more catch: rates changed last week, after training ended.
  11. You test the new model with a customer question.
  12. It gives last week's rate. What it learned froze on the day training ended.

Training first: the slowest, priciest path. Bloomberg trained BloombergGPT on 512 GPUs for about 53 days. A 2023 study found GPT-4, with prompts only, beat its published scores on 4 of 5 finance tests. And facts trained in don't update.

With the cheapest step first

  1. Same company, cheaper start: clear written instructions for an AI the company already has.
  2. Short rules plus a few example answers. This is a better prompt, and it's ready the same day.
  3. You test it with the same customer question.
  4. Before answering, the AI looks the rate up in the company's own pages.
  5. It answers with today's rate, and says where it found it.
  6. This is RAG (retrieval-augmented generation): look it up, then answer. When a rate changes, someone edits the page.
  7. Fine-tuning, training an existing model a bit more on your examples, comes last: for one narrow job, like tagging tickets.

Start cheap. Train last. Start with a better prompt. Look up facts that change (RAG). Fine-tune only for a narrow job or a fixed style. OpenAI's own guide says the prompt is usually the best place to start.

Keep going:

Why it matters, what it costs, and what to ask your team.

Three ways

A better prompt, look-ups, or training.

Your AI doesn't know your prices, your policies or your tone. You can teach it three ways, and each one costs more than the last.

Prompt: write better instructions, with examples. RAG (retrieval-augmented generation): let the AI search your documents before it answers. Fine-tuning: train an existing model a bit more on your own examples.

OpenAI's own guide says the prompt is usually the best place to start. Look-ups fix missing or outdated facts; fine-tuning fixes inconsistent style and format.3 Answer two questions to find your step on the ladder:

More cost and time as you climb ↑

  1. Train your own model$$$$

    Set up: months. BloombergGPT trained about 53 days on 512 GPUs.1Change a fact: train again

  2. Fine-tune$$$

    Set up: weeks. Collect, check, train, test.Change a fact: retrain

  3. Look-ups (RAG)$$

    Set up: days. Search over your documents.Change a fact: edit the document

  4. Better prompt$

    Set up: hours. Rules plus a few examples.Change a fact: edit the text

Setup times are typical ranges, not measurements. The top rung is outside the three: a whole new model.

Notice what fine-tuning is not for: facts that change. Facts trained into a model freeze on the day training ends.

So training our own model isn't step one?

It's usually the last step. Try a better prompt first.

The real case

Bloomberg trained its own. Then GPT-4 took the same tests.

In 2023, Bloomberg built BloombergGPT: a 50-billion-parameter model trained from scratch, about half on financial text from its own archives.1 That's beyond fine-tuning. It's a whole new model.

A study that year gave GPT-4, a general model, the same five public finance tests, using only prompts.2 Pick a test:

0billion parameters
0GPUs to train it
~0days of training

Scores (F1 or accuracy, 0 to 100, rounded) from Li et al., 2023. BloombergGPT's are the scores Bloomberg published; test setups were not always identical.

GPT-4 beat BloombergGPT's published scores on 4 of the 5 tests. The authors suggest the bigger size of GPT-4 may explain it.2

Now look at the grey bars. On two narrow tasks, a small model trained for just that task beat both big ones. That's where fine-tuning shines.

53 days of training, and a general model still won 4 of 5?

Ask your team

Three questions before anyone trains anything.

Before a training budget gets approved, ask:

  1. What exactly is the AI getting wrong?Missing or outdated facts point to look-ups. Wrong style or format points to a better prompt, then fine-tuning.
  2. How will we know it got better?Write a test set of real questions before changing anything. Score every option on the same set.
  3. What happens when a fact changes?With look-ups, someone edits a document. With a trained model, you may have to train again.

Under the hood, step by step: teach a help desk your business, cheapest step first.

Build along

Teach a help desk your business, one step at a time

We start with a plain question to a general model, score it on the same 30 real questions, and climb only when the score says so. Press Run it after each step and watch the result change.

The order of operations

Measure, then climb one rung at a time.

Every step starts and ends with the same test set. Climb only when the numbers say the cheaper rung isn't enough.

Write a test setreal questions, good answers Prompt + examples Good enough? yes Ship it no Add RAGfacts missing or out of date Fine-tunestyle or format still off Run the test set again
  1. Write the test set first30 or more real questions with good answers. Every option gets scored on the same set.
  2. Squeeze the promptClear rules, 3 to 5 examples, the exact output format. OpenAI calls the prompt the best place to start.3
  3. Facts go in documentsPrices, policies, anything that changes: put it in RAG, not in training data.
  4. Fine-tune small, then growOpenAI suggests starting with 50 well-crafted examples and measuring before adding more.5
  5. Re-test after every changeNew prompt, new documents, new model: run the same test set each time.

References

  1. "BloombergGPT: A Large Language Model for Finance" · Wu et al., arXiv, Mar 30, 2023. 50 billion parameters, about half financial data, ~53 days on 512 GPUs, published scores on five public finance tests.
  2. "Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks" · Li et al., arXiv, May 10, 2023; EMNLP 2023 Industry Track. GPT-4 vs BloombergGPT and small fine-tuned models.
  3. "Optimizing LLM Accuracy" · OpenAI API docs. Start with the prompt; RAG for missing or outdated knowledge; fine-tuning for consistent behavior.
  4. "Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs" · Ovadia et al., arXiv, Dec 10, 2023. RAG consistently beat unsupervised fine-tuning for adding facts.
  5. "Supervised fine-tuning" · OpenAI API docs. Chat-format JSONL training data; start with 50 well-crafted examples.

The company, its rates, the chats, the screens, the decision helper and the build-along's code and scores are made up to show the idea. Setup times are typical ranges, not measurements. BloombergGPT's numbers and all benchmark scores come from the cited papers.

Made by Abed Manzalji, AI Engineer in Toronto.More about meContact