How ChatGPT Is Trained: Pretraining, Fine-Tuning and RLHF
Learn how ChatGPT is trained in three stages: pretraining on huge text, fine-tuning on written examples, and RLHF. Plain English, with takeaways for founders.
If you build with AI, it helps to know how ChatGPT is trained. Each stage changes the model in a different way, and each one tells you something about what the model can and cannot do for your product. This post walks through the three stages in plain words, then shows what they mean for a founder choosing between prompting, retrieval and fine-tuning.
One note first. Labs do not publish every detail of how their current models are trained, so this post describes the general recipe and the parts that have been written down in public, not any company's exact process.
How ChatGPT is trained, in three stages
Think of it as three steps, each building on the last.
- Pretraining. The model reads a huge amount of text and learns to predict what comes next. The result is a raw model that is good at continuing text.
- Supervised fine-tuning. People write examples of good replies. The model is trained on them so it behaves like an assistant instead of a text-continuer.
- RLHF (reinforcement learning from human feedback). People compare answers and say which one they prefer. The model is nudged toward the answers people rank higher.
This matches what is written down in public. Wikipedia's page on ChatGPT says it started from GPT foundation models adapted with supervised learning, where trainers wrote sample exchanges, and then with RLHF, where trainers ranked the model's responses and those rankings trained reward models.
Stage 1: pretraining on internet-scale text
Pretraining is the biggest and most expensive stage. The model is shown enormous amounts of text and given one simple job: guess the next piece of text. The pieces are called tokens, and a token is often a word or part of a word.
Take the sentence "The capital of France is". A model that has seen enough text learns that "Paris" is a very likely next token. When it guesses wrong, its internal numbers are adjusted slightly. Repeat that across a vast amount of text and the model gets very good at the patterns of language.
For the mechanics in more depth, read how large language models work through next-token prediction. The longer story is in a short history of AI from Turing to ChatGPT.
What pretraining teaches the model, and what it does not
To predict text well, a model has to pick up a lot along the way: grammar, facts that show up often, styles of writing, even some code. That is why a pretrained model feels so capable.
But a pretrained model is not an assistant. Ask it "What is a good name for a bakery?" and it might continue with another question, or a list of similar questions, because that is a plausible way for text to go on. It was trained to continue documents, not to answer you.
Pretraining also does not give the model a way to check facts. It learns what is likely to be said, not what is true. This is the root of the confident wrong answers you will see in practice. The Basic course How Modern AI Actually Works in AI Entrepreneur has a lesson on it, "What AI Gets Wrong".
Stage 2: supervised fine-tuning on written examples
The second stage turns a text-continuer into something that follows instructions. People write prompts and the replies they would want to see. The model is then trained on those pairs, using the same next-token idea as before, just on a much smaller and more carefully chosen set of text.
An example pair, written by us for this post:
Prompt: Explain what a landing page is to a first-time founder in two sentences. Written reply: A landing page is a single web page built to get one action from a visitor, such as joining a waitlist. It lets you test whether people want your product before you build it.
After enough pairs like this, the model learns the shape of a good reply: answer the question, stay at the right length, follow the format requested. The knowledge still comes from pretraining. Fine-tuning mostly teaches behavior.
Stage 3: RLHF, teaching the model what people prefer
Writing perfect answers by hand is slow. Judging which of two answers is better is much faster. RLHF uses that fact. In outline:
- The model produces several answers to the same prompt.
- People rank them from best to worst.
- Those rankings train a separate scoring model, called a reward model, that learns to predict which answers people will prefer.
- The main model is then adjusted, using reinforcement learning, to produce answers that score higher.
RLHF has a limit. The model is tuned toward what raters prefer, and what raters like is not always what is correct. A fluent, confident answer can win over a correct but hedged one. So RLHF can make a model feel more helpful without making it reliably right. That gap is why a product that matters needs its own checks on output quality.
What the InstructGPT paper found
The clearest public write-up of this recipe is a 2022 paper, "Training language models to follow instructions with human feedback" (Ouyang et al., arXiv, 4 March 2022). It describes the same two steps on top of GPT-3: fine-tune on demonstrations written by labelers, then use RLHF on rankings of model outputs. The resulting models are called InstructGPT.
The authors report that human evaluators preferred outputs from InstructGPT over those from the much larger GPT-3. Their opening point is that making a language model bigger does not, by itself, make it better at following what a user wants. Those are the authors' findings on their own models.
The AI Entrepreneur lesson "From Raw Model to Helpful Assistant" is about that same step.
Why training costs so much, and what it means for startups
Pretraining is the expensive part. It needs a very large amount of data, specialized hardware running for a long time, and a team to manage it.
For a first-time founder, the practical conclusion is simple: you will almost certainly not pretrain a model. You will rent one through an API and pay per request. Your costs then depend on how much text you send and receive, which makes unit economics a day-one concern. The lesson "Model APIs, Tokens and What a Request Costs" in the Intermediate course The AI Product Stack is about what a single request costs.
If everyone can rent the same model, the model alone is not what makes you different. That is the heart of is my AI startup just a thin wrapper around ChatGPT. Your edge has to come from elsewhere: the problem you understand, the data you can use, the way you reach customers.
Prompting, retrieval or fine-tuning: what founders should do
You have three main ways to make a model work better for your use case. Try them in this order.
1. Prompting
Write clear instructions, give examples, and state the format you want. It costs nothing to try and you can change it in seconds. A made-up example for a small tool that rewrites customer emails:
You rewrite customer support emails. Keep a friendly, direct tone. Never promise refunds. Output the reply only, under 120 words. Here are two examples of good replies: ...
2. Retrieval (RAG)
If the model needs facts it was not trained on, such as your product docs or a customer's own files, do not try to teach it those facts. Look up the relevant text when the request comes in and put it in the prompt. This keeps answers current and lets you point to a source. The lesson "Giving Models Your Data: Retrieval" is about the idea.
3. Fine-tuning
Fine-tuning makes sense when you need a steady style, format or behavior that prompting cannot hold, and you have a good set of examples. It is about behavior, not about adding a pile of new facts. Before you commit, you need a way to measure whether the tuned model is actually better. That is what evals are for, and the Advanced course Evals: Shipping AI That Works has a lesson called "Building an Eval Set and Judging".
A good rule: start with the smallest version one that uses prompting. Move to retrieval when you hit a knowledge gap. Consider fine-tuning only after you have real usage, real failures and a way to score them. Choosing between big, small, open and closed models is a related decision, in the lesson "Choosing Models: Big, Small, Open, Closed".
Frequently asked questions
What are the three stages of training ChatGPT?
Pretraining teaches the model to predict the next token from a huge body of text. Supervised fine-tuning trains it on human-written examples so it behaves like an assistant. RLHF then adjusts it toward answers that people rank as better.
What does RLHF stand for?
It stands for reinforcement learning from human feedback. People compare model answers, and those comparisons are used to push the model toward responses people prefer.
Can a startup train its own model like ChatGPT?
Pretraining a large model from scratch is very costly, so most startups rent a model through an API instead. Try prompting and retrieval first, and consider fine-tuning only when you can measure the result.
Does RLHF make the model always correct?
No. It makes answers more helpful and better matched to what raters prefer, but a fluent answer can still be wrong. You still need to check output quality in your own product.
Is fine-tuning the way to teach a model my company's facts?
Usually retrieval is the better fit for facts, because you can keep the source text up to date and show where an answer came from. Fine-tuning is better suited to steady style and behavior.
Get started
The Basic courses need no background, and each lesson takes 8 to 12 minutes. Every lesson is free. Start learning with AI Entrepreneur: sign in on the home page and take your first short lesson today.