Quantization
The model is too big to fit / serve.
The idea inside
Store weights at lower precision; small accuracy cost, big memory win.
After this lesson
You can explain quantization: fewer bits per weight, big memory win, small quality cost.
Where it leads
How far has this same machine been scaled?
Inside this lesson
That's the real lesson stage, paused. Claim your pass to operate it.
See how AI actually works, end to end.
This lesson is one stop on the full arc. Unlock all of it, and keep it for life.
What you get
- The 34-lesson main path, a finishable route from a word to agents
- Goal tracks for using AI at work and building AI features
- Boss labs that make you apply a whole act, not just recognize it
- Spaced recall that brings each idea back before you forget
- Course memory: every term defined, with links to where it first appears
- A shareable capability card when you finish the main path
- Lifetime access on every device, every future lesson included
Not videos to watch. You predict, operate the machine, then prove it. That is why it stays.
99 interactive lessons and challenges. No videos, no code.
Free launch pass: lifetime access, no card needed
New here? The first lessons are free to try. Start with lesson 0.1
What this lesson shows
Store weights at lower precision; small accuracy cost, big memory win.
The question it opens with
The model is too big to fit / serve.
The walkthrough, in the lesson's own words
- Drag to make the ruler coarser. Watch the size drop and the quality hold.
- Here's the trick: round every weight to the nearest mark on a coarser ruler.
- Now you drive. Make the ruler coarser and watch the size and the quality.
- The quality-kept percentages are invented to show the trend; real numbers vary by model and task.
- Coarser ruler, fewer digits per weight: round 0.7341892 to 0.73 and you store almost the same number in a fraction of the space, and the quality barely moves.
- Models actually ship at 16 bits per weight (BF16): 70 billion weights × 16 bits ÷ 8 = 140 GB (÷ 8 because 8 bits = 1 byte), still too big for one ~80 GB chip. The 280 GB FP32 stop is the old full-precision reference; nobody serves a model at FP32 today.
- Each weight drops to the nearest tick. A little off, but barely.
- The rounded value is a little off, but a dot product (the multiply-and-add from 1.3) only needs roughly-right values, so the model's decisions barely move.
- One catch: a few unusually large "outlier" weights carry outsized influence and crack under naive 4-bit rounding, so real quantizers calibrate around them first, or retrain with the rounding in the loop (quantization-aware training).
- Fewer bits is like fewer colors in a photo: the file shrinks a lot, the picture barely changes… until it does. That's
- Same mental model, new names in 2026: models are trained and shipped in BF16 (16 bits), most serving runs them at FP8 (8 bits), and FP4 formats (NVFP4, MXFP4) are arriving with the newest chips. Each rung is the move you just made, half the bits, almost the same answers.
- You can run a big model locally on your laptop, and it's smaller and faster but answers a touch worse than the cloud version. What did the local copy trade away?
- A local build is usually quantized: each weight is stored in fewer bits (say INT4 instead of FP16), which shrinks the file and speeds it up with only a small quality cost. That trade is great for everyday use, but for tasks that need every last bit of accuracy, reach for the higher-precision version.
Key takeaway
Shrink the model, mostly keep the smarts.
What you can do after this lesson
You can explain quantization: fewer bits per weight, big memory win, small quality cost.
Prefer reading first? This lesson has a companion explainer: Quantization
Where it leads: How far has this same machine been scaled?
This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.