Plain-language explainer
The KV cache, explained
What is the KV cache, and why does it matter for speed and cost?
When a model generates text, each new token needs to look back at every earlier token through attention. Recomputing that look-back from scratch for every token would be brutally slow. The KV cache stores each token's attention keys and values the first time they are computed, so every later step reuses them and only computes the newest token. The price is memory: the cache grows with context length, which is why long conversations eat GPU memory.
Do not just read it. Operate the mechanism yourself in a short interactive lesson.
See it work: Context window & KV cache →Free, no code, no signup.
What people get wrong
- The KV cache is the model's memory of you. It is per-request working memory, thrown away after the response, and it never changes the model.
- The cache stores your text. It stores attention keys and values, the vectors the model derived from your text.
- A longer context window is just a bigger text box. Every token's keys and values must sit in GPU memory at once, so context length is a hardware budget.
Where you see it in real products
- Very long chats slow down and hit limits because the cache keeps growing.
- API providers sell cheaper cached input tokens: the keys and values for a repeated prompt prefix already exist.
- Context-length specs on model cards are as much about GPU memory as about the model itself.
Related explainers
Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.