RLVR: how AI got good at math and code

What this lesson shows

A third training stage: reinforcement learning on rewards a program can check, did the code run, is the answer right, instead of a human thumbs-up. The model becomes whatever its reward can measure.

The question it opens with

Models jumped from mediocre to expert at math and code in barely a year, with no new architecture. What changed?

The walkthrough, in the lesson's own words

• Same brain, a year later, expert at math and code. Predict what changed.
• You're the checker. Run each attempt and see which reasoning gets rewarded.
• A checker can only grade what a program can score. Where does that leave writing?
• Reward what a program can check, and the model becomes good at it.
• Same transformer. No new architecture. So what changed?
• What made models jump from mediocre to expert at math and code?
• Right. Not a bigger or new-shaped model, a new training stage: reward the model for answers a program can check.
• It was the training, not the model. A new stage rewards the model for answers a program can check, not a human thumbs-up.
• You've seen two stages: pretraining (predict the next word, Act 2) and post-training with human feedback (SFT + RLHF, 2.6). RLVR is a third stage. RL stands for reinforcement learning, and VR for verifiable rewards: instead of a human thumbs-up, the reward comes from a program that CHECKS the answer, did the code run, is the math right. Turn that reward loose on math and code, where correctness is checkable, and the model climbs fast.
• RLVR made models better at everything, including writing and open questions.
• RLVR only trains what a program can grade: math, code, formal tasks. There's no checker for a good poem or a tactful email, so open-ended writing still leans on human-feedback tuning (RLHF, E.1), not verifiable rewards.
• The whole trick is a reward the model can't fake: a deterministic program compares its answer to a known-correct one. That only exists where correctness is objective. For a poem or an email there's no ground truth to check against, so RLVR has nothing to reward, and labs fall back to human raters or an AI judge.
• A thumbs-up rewards confident flattery. An answer checker rewards trying again until it's right, and long self-correcting reasoning falls out.
• DeepSeek-R1 (early 2025) showed this at scale: train a base model with RL on verifiable math and code rewards, using a recipe called GRPO (group relative policy optimization), and long reasoning chains emerge on their own, no human wrote them. The model learns to pause, re-check, and correct itself, because that's what earns the reward. That's the same lineage as the reasoning models in 4.8.
• A new model is far better at coding contests but no better at writing a heartfelt eulogy. Given how it was trained, why the split?
• RLVR rewards only what a program can check. Coding has a checker (does it run, does it pass tests?), so verifiable rewards trained it hard there. A eulogy has no ground truth to grade against, so that skill still comes from human-feedback tuning, which improves much more slowly and can't be scaled the same way.
• The math problem, attempts, and grades are invented to show the mechanism, not logged from a real training run.
• Only the attempt that stopped, re-checked, and fixed itself earned a reward. Do that a million times and 'pause and verify' becomes the model's default, that's where long reasoning chains come from.

Key takeaway

You played the verifier: a thumbs-up rewards confident flattery, an answer checker rewards trying again until it's right, and long self-correcting reasoning falls out.

What you can do after this lesson

You can explain why models suddenly got good at checkable domains: verifiable rewards let training scale wherever a program can grade the answer, and flattery stops paying.

This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.