What does RLHF actually change about a model?

Not what it knows, but what it tends to say. A pretrained model continues text; RLHF tunes it toward the kind of continuation people rated as helpful and safe. Same knowledge, very different default behaviour.

Where do the human ratings come from?

Paid annotators comparing pairs of answers to the same prompt against a written guideline. Those comparisons train a reward model, which then scores far more answers than humans ever could. That guideline is effectively the product's values, written down.

Is RLHF why models refuse things?

Mostly yes, together with the safety rules layered on top. Refusals and hedging are learned behaviours that were rewarded, which is also why they get over-applied to harmless questions that merely resemble risky ones.

Plain-language explainer

RLHF, explained

What is RLHF, and why does a text predictor act like an assistant?

Pretraining produces a raw predictor that continues text in any direction, helpful or not. RLHF is the finishing school: humans compare pairs of model answers and pick the better one, a reward model learns to imitate those preferences, and the LLM is then trained to score highly on that reward. The result answers questions, follows instructions and declines harmful requests, not because it was told rules, but because responses shaped that way rated better.

Last reviewed July 25, 2026

Reading is the slow way. Start with a free lesson you can operate right now.

Start free: From the internet to your answer →

Free, no code, no signup.

Then go deeper: RLHF / post-training Locked

What people get wrong

RLHF teaches the model facts. Knowledge comes from pretraining. RLHF shapes behavior, tone and refusals.
The assistant has values it reasons from. It has a policy optimized to produce answers humans preferred.
RLHF makes models more truthful. It can reward pleasing over accurate, which is where sycophancy comes from.

Where you see it in real products

The thumbs up and down buttons collect exactly this preference data.
'Which response do you prefer?' side-by-side prompts in chat apps feed the reward model.
The gap between a raw base model and the polite assistant you use is mostly this step.

Common questions

What does RLHF actually change about a model?: Not what it knows, but what it tends to say. A pretrained model continues text; RLHF tunes it toward the kind of continuation people rated as helpful and safe. Same knowledge, very different default behaviour.
Where do the human ratings come from?: Paid annotators comparing pairs of answers to the same prompt against a written guideline. Those comparisons train a reward model, which then scores far more answers than humans ever could. That guideline is effectively the product's values, written down.
Is RLHF why models refuse things?: Mostly yes, together with the safety rules layered on top. Refusals and hedging are learned behaviours that were rewarded, which is also why they get over-applied to harmless questions that merely resemble risky ones.

Related explainers

Fine-tuning vs prompting Next-token prediction Transformers LLM limits

One idea at a time, in your inbox

New lessons and explainers, written the way these pages are. Now and then, not daily, and never a sales sequence.

We email you a confirmation link first. Unsubscribe in one click, any time. Privacy.

Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.