How a pile of internet text becomes something that answers your prompts — and where “fine-tuning” and “RL” actually live.
Your mission: build AI products better by understanding how the models are made. This lesson is the map. Every later lesson is a zoom-in on one box.
You use LLM APIs every day. When you call one, you’re talking to a model that went through three distinct stages of training. Almost everything that puzzles people about model behavior — why it’s helpful, why it refuses, why one model “feels” better than another at the same size — traces back to which stage did what.
Here’s the whole pipeline on one screen. Hold this picture; the rest of the course is footnotes to it.
Stage 1 · Pretraining
Learn to predict the next token
→ gives a base model: a vast autocomplete engine that knows the world but isn’t an assistant.
↓
Stage 2 · Supervised Fine-Tuning (SFT)
Imitate good example answers
→ gives an instruction-following model: it now answers questions instead of just continuing text.
↓
Stage 3 · Preference / RL Tuning (RLHF & friends)
Optimize toward what humans prefer
→ gives the polished assistant you actually ship against: helpful, harmless, on-tone.
Stages 2 and 3 together are called post-training. That single word is the thing you asked about. Pretraining builds raw capability; post-training shapes that capability into a usable, well-behaved product. 1
Stage 1 — Pretraining: the autocomplete brain
A base model is trained on a huge slice of the internet with one absurdly simple objective: predict the next token (roughly, the next word-piece) over and over, billions of times. That’s it. No questions, no answers, no “assistant.” Just: given this text so far, what comes next?2
Analogy
A base model is like a developer who has read every repo on GitHub but has never been told what their job is. Stupendous latent knowledge; zero sense of “when someone asks me X, I should do Y.” Ask a raw base model “What is the capital of France?” and it might reply with another list of trivia questions — because on the internet, that string is often followed by more questions, not an answer.
So the base model is not broken — it’s doing exactly what it was trained to do. It just wasn’t trained to be helpful. That’s the gap post-training fills.
To turn the autocomplete brain into something that answers, we keep training the same model, but now on a curated set of example conversations: a prompt, paired with an ideal response written (or vetted) by humans. The model learns by imitation — “when the input looks like a question, produce text that looks like this good answer.”3
Analogy
This is onboarding by worked examples. You hand the well-read new hire a few thousand “here’s a good ticket, here’s how a senior engineer responded” pairs. They’re not learning new facts so much as learning the job: the shape, tone, and helpfulness expected of a reply.
Key term“Fine-tuning” just means continuing to train an existing model on a smaller, targeted dataset. SFT is the most important fine-tuning step in post-training — but “fine-tuning” is a general technique, not a single specific thing. (This is the word that trips most people up; we’ll nail it in the wrap-up.)
Stage 3 — Preference / RL tuning: optimize toward what people prefer
SFT gets you a model that answers. But “an answer” and “the answer a human would prefer” are different bars. There are usually many acceptable replies, and writing the single ideal answer for every situation by hand doesn’t scale. It turns out humans are far better at comparing two answers than at writing the perfect one.4
So the classic recipe — RLHF, Reinforcement Learning from Human Feedback — does this:
Collect preferences. Show humans two model answers to the same prompt; they pick the better one. Do this a lot.
Train a reward model. A second model learns to predict those human judgments — it outputs a score for “how much would a human like this answer?”
Optimize the assistant against that score using reinforcement learning: nudge the model to produce answers the reward model rates highly.1
Analogy
SFT was learning from worked examples. RLHF is code review at scale: the model proposes answers, a reviewer (the reward model, trained on real human taste) scores them, and the model gradually adjusts to write what gets approved. It’s feedback on its own attempts, not copying someone else’s.
So what is the “RL” people mean?Reinforcement learning = learning by trying things and being rewarded or penalized, rather than by copying labeled examples. In post-training, “RL” almost always refers to this Stage-3 preference optimization (RLHF, or newer variants). When someone says a model was “aligned with RLHF,” they mean Stage 3 happened. (Newer methods like DPO reach a similar goal without a separate reward model and full RL loop — a later lesson.)5
Untangling the two words you came in with
Let’s settle it directly, because the field uses these loosely:
Fine-tuning — the broad technique of further-training an existing model on focused data. SFT (Stage 2) is the headline example, but the word also gets used loosely for “any post-training” and for your-own customization of a model.
RL / RLHF — a specific Stage-3 method for shaping behavior via preference feedback instead of copied examples. It’s a kind of post-training, not a synonym for it.
Post-training — the umbrella: everything after pretraining (Stages 2 + 3) that turns a base model into a shippable assistant.
This map is a decision tool. When a model misbehaves or you’re choosing one, you can now ask which stage owns the problem:
“It doesn’t know my company’s docs.” That’s a knowledge gap → usually solved by giving it context (RAG / prompting), not by RL. The base model’s knowledge is frozen at pretraining.
“It knows the facts but won’t follow my format/tone.” That’s behavior → an SFT-style fine-tune, or just better prompting, is the lever.
“Why does Model A feel so much more helpful than Model B at the same size?” Often a post-training difference (better SFT data + RLHF), not raw capability. Two models with similar pretraining can feel worlds apart after Stage 3.
You’ll spend the rest of this course making each of these judgments sharper.
Retrieval practice
Don’t re-read to answer — try to recall. Effortful retrieval is what makes this stick. Immediate feedback below each.
1. What does pretraining alone produce?
A base model just predicts the next token — enormous knowledge, but no instinct to be a helpful assistant. That instinct comes from post-training.
2. Supervised fine-tuning (SFT) teaches the model mainly by…
SFT is imitation learning: prompt + ideal response pairs. The model learns the *job* (how to answer), not new facts. Ranking and scoring belong to Stage 3 (RLHF).
3. RLHF optimizes the assistant against…
Humans compare answers → a reward model learns that taste → the assistant is tuned (via RL) to score well. Labeled examples are SFT; unlabeled corpora are pretraining.
Primary source to go deeper
Watch the first ~half of Andrej Karpathy — “Deep Dive into LLMs like ChatGPT.” It walks the exact pipeline above, with visuals, from someone who built these systems. The best concepts-first source we found.2
Your teacher is on call. I’m the agent that wrote this lesson — ask me anything that’s fuzzy. Good follow-ups for this one: “Walk me through what a reward model actually outputs,” or “If I just want it to use my docs, which stage am I touching?” Or just say “next lesson.”