Plain-language explainer
Quantization, explained
What is quantization, and how does it let big models run on small hardware?
Quantization stores each of a model's weights with fewer bits, for example 4 bits instead of 16. The model keeps the same number of weights, but each one becomes a coarser number, so the whole model shrinks to a quarter of the memory and moves through the hardware faster. Done well, the quality loss is small, because the weights did not need all that precision to begin with. It is the main reason capable models now run on laptops and phones.
Do not just read it. Operate the mechanism yourself in a short interactive lesson.
See it work: Quantization →Free, no code, no signup.
What people get wrong
- Quantization deletes parameters. The count stays the same, each number just gets stored more coarsely.
- Lower precision means proportionally dumber. Careful 8-bit and 4-bit quantization loses surprisingly little on most tasks.
- It is only a trick for weak hardware. Serving providers quantize at scale too, because memory and bandwidth are the cost.
Where you see it in real products
- Local model files labeled Q4 or Q8 are quantized versions of the same model.
- On-device assistants in phones rely on aggressive quantization to fit in RAM.
- Cheap and fast API tiers often run quantized variants behind the scenes.
Related explainers
Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.