Does quantization make a model worse?

A little, and usually less than people expect. Going from 16-bit to 8-bit is often hard to notice, 4-bit is a real but frequently acceptable trade, and below that quality falls off quickly. Compare on your own task, not on a general benchmark.

Why does quantization make generation faster?

Generation is mostly limited by moving weights out of memory, not by the arithmetic. Halving the bytes per weight roughly halves that traffic, so the same hardware produces tokens faster and can hold a larger model at all.

What does Q4 or 4-bit in a model filename mean?

The number of bits used per weight. Fewer bits means a smaller file and lower memory use at some cost in fidelity. The letters and numbers after it identify the specific quantization scheme, which differ mainly in how they handle unusually large weights.

Plain-language explainer

Quantization, explained

What is quantization, and how does it let big models run on small hardware?

Quantization stores each of a model's weights with fewer bits, for example 4 bits instead of 16. The model keeps the same number of weights, but each one becomes a coarser number, so the whole model shrinks to a quarter of the memory and moves through the hardware faster. Done well, the quality loss is small, because the weights did not need all that precision to begin with. It is the main reason capable models now run on laptops and phones.

Last reviewed July 25, 2026

Reading is the slow way. Start with a free lesson you can operate right now.

Start free: Why GPUs beat CPUs →

Free, no code, no signup.

Then go deeper: Quantization Locked

What people get wrong

Quantization deletes parameters. The count stays the same, each number just gets stored more coarsely.
Lower precision means proportionally dumber. Careful 8-bit and 4-bit quantization loses surprisingly little on most tasks.
It is only a trick for weak hardware. Serving providers quantize at scale too, because memory and bandwidth are the cost.

Where you see it in real products

Local model files labeled Q4 or Q8 are quantized versions of the same model.
On-device assistants in phones rely on aggressive quantization to fit in RAM.
Cheap and fast API tiers often run quantized variants behind the scenes.

Common questions

Does quantization make a model worse?: A little, and usually less than people expect. Going from 16-bit to 8-bit is often hard to notice, 4-bit is a real but frequently acceptable trade, and below that quality falls off quickly. Compare on your own task, not on a general benchmark.
Why does quantization make generation faster?: Generation is mostly limited by moving weights out of memory, not by the arithmetic. Halving the bytes per weight roughly halves that traffic, so the same hardware produces tokens faster and can hold a larger model at all.
What does Q4 or 4-bit in a model filename mean?: The number of bits used per weight. Fewer bits means a smaller file and lower memory use at some cost in fidelity. The letters and numbers after it identify the specific quantization scheme, which differ mainly in how they handle unusually large weights.

Related explainers

GPUs for AI Mixture of Experts How LLMs work Distillation

One idea at a time, in your inbox

New lessons and explainers, written the way these pages are. Now and then, not daily, and never a sales sequence.

We email you a confirmation link first. Unsubscribe in one click, any time. Privacy.

Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.