Plain-language explainer
The KV cache, explained
The KV cache is the memory a language model keeps during generation: the attention Keys and Values of every token processed so far. Storing them means each new token only computes its own row instead of recomputing the whole history, which makes generation fast. Its size grows with context length, which is why long contexts eat GPU memory.
What is the KV cache, and why does it matter for speed and cost?
When a model generates text, each new token needs to look back at every earlier token through attention. Recomputing that look-back from scratch for every token would be brutally slow. The KV cache stores each token's attention keys and values the first time they are computed, so every later step reuses them and only computes the newest token. The price is memory: the cache grows with context length, which is why long conversations eat GPU memory.
Last reviewed
Reading is the slow way. Start with a free lesson you can operate right now.
Start free: Predict the next word →Free, no code, no signup.
Ten seconds, no lesson: play with the standalone widget, and embed it in your own page.
Then go deeper: Context window & KV cache Locked
How the KV cache works, step by step
Inside every attention layer, each token produces three vectors: a Query, a Key and a Value. To write the next token, the model compares that token's Query with the Key of every earlier token, then blends their Values. The Query is only needed once. The Keys and Values of earlier tokens are needed again at every later step.
Those earlier Keys and Values never change. A token can only look backwards, so nothing written later alters what an earlier token computed. That makes them safe to store. The KV cache is that store: one Key and one Value per token, per layer, per attention head, kept in GPU memory for the length of the request.
- Prefill: the whole prompt is processed in one parallel pass, and every prompt token's Keys and Values are written to the cache. This is the wait before the first word appears.
- Decode: each new token computes only its own Query, Key and Value, reads everything else from the cache, and appends its own Key and Value. One token per step.
- Release: when the response ends, the cache is freed, unless the provider keeps the prefix for prompt caching.
How much memory does the KV cache use? A worked example
The size follows one formula: 2 (one Key and one Value) times the number of layers, times the number of key-value heads, times the size of each head, times the bytes per number, times the number of tokens.
Take Llama 3.1 8B: 32 layers, 8 key-value heads, 128 numbers per head, stored at 2 bytes each. That is 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KB for every token. At 8,192 tokens the cache is 1 GB. At the full 128,000-token context it is about 16 GB, roughly the size of the model's own weights at the same precision.
The older Llama 2 7B gave every attention head its own Keys and Values, 32 per layer instead of 8, so it needed 512 KB per token, four times more. Llama 3.1 70B, with 80 layers, needs 320 KB per token, so one full-length conversation holds about 40 GB of cache. Multiply by the number of people a server handles at once and the cache, not the model, is often what fills the GPU.
| Model | Per token | At 4K tokens | At 128K tokens |
|---|---|---|---|
| Llama 2 7B (32 KV heads) | 512 KB | 2 GB | 64 GB, past its 4K limit |
| Llama 3.1 8B (8 KV heads) | 128 KB | 0.5 GB | 16 GB |
| Llama 3.1 70B (8 KV heads) | 320 KB | 1.25 GB | 40 GB |
Why the KV cache matters in real models
Most recent architecture changes in attention exist to shrink this cache. Grouped-query attention, used by Llama 2 70B, Llama 3 and Mistral, lets several Query heads share one set of Keys and Values. That is where the 4x saving above comes from. DeepSeek-V2 went further with multi-head latent attention, which stores a compressed version of the Keys and Values, and its paper reports a 93% smaller cache than DeepSeek's earlier 67B model.
Serving software is built around it too. vLLM's PagedAttention stores the cache in fixed-size blocks, like pages of computer memory. Its authors measured that earlier systems wasted 60 to 80% of cache memory on gaps and reservations, and that paging cut the waste below 4%, which let one GPU serve two to four times as many requests. Many servers also store the cache at 8 bits instead of 16 to fit twice the context.
And you see it on your bill. Prompt caching keeps the cache of a repeated prefix, like a long system prompt or a document, alive between requests. Anthropic bills those cached reads at a tenth of the normal input price, and OpenAI discounts repeated prefixes automatically. The discount is the cache you did not have to rebuild.
KV cache vs prompt caching vs context window
These three are related and often mixed up. The context window is the limit on how many tokens a model can consider. The KV cache is the working memory that holds those tokens' Keys and Values during one request. Prompt caching is a product feature that keeps part of that memory between requests.
| What it is | How long it lives | Who controls it | |
|---|---|---|---|
| Context window | The maximum number of tokens per request | A fixed property of the model | The model's design |
| KV cache | Stored Keys and Values for the tokens so far | One request | The serving software, automatically |
| Prompt caching | A KV cache for a repeated prefix, kept between requests | Minutes, sometimes hours | You, through how you structure prompts |
What people get wrong
- The KV cache is the model's memory of you. It is per-request working memory, thrown away after the response, and it never changes the model.
- The cache stores your text. It stores attention keys and values, the vectors the model derived from your text.
- A longer context window is just a bigger text box. Every token's keys and values must sit in GPU memory at once, so context length is a hardware budget.
Where you see it in real products
- Very long chats slow down and hit limits because the cache keeps growing.
- API providers sell cheaper cached input tokens: the keys and values for a repeated prompt prefix already exist.
- Context-length specs on model cards are as much about GPU memory as about the model itself.
Common questions
- What does the KV cache actually store?
- The key and value vectors already computed for every token in the context. Without it, generating token 500 would mean reprocessing the previous 499 from scratch. With it, only the newest token is processed and the rest is read back.
- Why does the first token take so much longer than the rest?
- That is prefill: the whole prompt is processed at once to build the cache. Everything after it is a cheap single-token step reading from that cache. It is why providers quote time-to-first-token and tokens-per-second as two separate numbers.
- Why does long context use so much memory?
- The cache grows with every token and has to stay in fast memory for the whole request. On a long context it can rival the model weights themselves, which is why long-context requests cost more and why shrinking the cache is an active research area.
- How big does the KV cache get?
- It scales with tokens times layers times the size of the keys and values. For a large model at a long context, that is gigabytes per conversation, which is why serving long chats needs so much GPU memory and why grouped-query attention was invented to shrink it.
- Is prompt caching the same as the KV cache?
- Prompt caching is a product feature built on the KV cache. When a provider keeps the keys and values of a prompt prefix in memory between requests, it skips recomputing them and charges you less for those cached input tokens. The mechanism underneath is the same cache.
- Does the KV cache change the model's answer?
- No. Reusing stored Keys and Values gives the same result as recomputing them, apart from tiny rounding differences. The cache only saves work. What changes the answer is shrinking it with lossy tricks, such as dropping old tokens or storing it at very low precision, which is why those are tested carefully.
Related explainers
More in Speed, cost and control
- What is model routing, and how do you choose which AI model to use?
- What is quantization, and how does it let big models run on small hardware?
- What does the temperature setting actually do to an AI model?
- Why are GPUs, and not CPUs, the hardware of the AI boom?
- How do reasoning models actually work?
- Why is running AI so expensive, and what are you paying for?
One idea at a time, in your inbox
New lessons and explainers, written the way these pages are. Now and then, not daily, and never a sales sequence.
We email you a confirmation link first. Unsubscribe in one click, any time. Privacy.
Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.