RLHF / post-training

What this lesson shows

Pretraining → SFT → RLHF: rank outputs to train a reward model, shift the policy.

The question it opens with

A raw pretrained model isn't helpful or aligned.

The walkthrough, in the lesson's own words

• Both replies are fluent. Tap the one you'd rather get back.
• You're the human rater. Pick the better reply in each pair.
• Your picks trained a judge. Now the judge grades replies you never saw.
• Click “Forgot password” on the sign-in page, then follow the reset link in your email.
• Tap the other reply to compare both fluent outcomes.
• Both are fluent, but only one is useful. A raw model can't tell them apart, that's the sense we have to teach it.
• Pretraining (Act 2) only taught the model to predict fluent text, never which reply is better. We have to teach that, and the rest of this lesson shows how.
• SFT (2.6) already taught this model to answer in an assistant voice. Ranking replies is the next signal.
• RLHF. Reinforcement Learning from Human Feedback: your handful of picks trained a judge (a reward model), and the judge, not you, grades millions of answers into the model's instincts.
• RLHF started this, but labs now mix a family of methods, all built on the same idea, prefer better replies:
• DPO (direct preference optimization): skip the separate judge and nudge the model directly on preference pairs. Cheaper, same signal.
• RLAIF: an AI rates the replies instead of people, so labs can grade millions of pairs.
• RLVR (verifiable rewards): for math and code, a program checks the answer, so the reward is exact. Trained this way (a recipe called GRPO), models learn to write long chains of thought. That's where the reasoning models in 4.8 come from.
• A chat assistant gets noticeably warmer and more cautious after an update, with no change to the questions you ask. What happened?
• RLHF reshaped its instincts: human raters' preferences trained a reward model, and that judge graded millions of answers to favor that tone. So when an assistant's style or refusals shift, the post-training and the raters behind it changed, not the underlying facts it was trained on.
• Your picks train a second model, the judge, called a reward model, to predict which reply a human would prefer; then the judge grades replies you never saw:
• 2 pairs here; real labs collect hundreds of thousands of comparisons.
• That last high score is the catch: the judge learned 'confident and friendly', not 'true'. A model chasing those scores learns to flatter instead of help. That failure is called reward hacking; the flattery flavor is sycophancy.
• Your handful of picks trained a judge. The judge, not you, grades millions of answers. Each score is a reward, gradient descent (2.4) nudges the weights to make high-scoring replies more likely.

Key takeaway

See how a handful of preferences becomes the judge that shapes the assistant.

What you can do after this lesson

You can explain RLHF: human preferences train a reward model that shapes the assistant.

Prefer reading first? This lesson has a companion explainer: RLHF

This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.