What does the KV cache actually store?

The key and value vectors already computed for every token in the context. Without it, generating token 500 would mean reprocessing the previous 499 from scratch. With it, only the newest token is processed and the rest is read back.

Why does the first token take so much longer than the rest?

That is prefill: the whole prompt is processed at once to build the cache. Everything after it is a cheap single-token step reading from that cache. It is why providers quote time-to-first-token and tokens-per-second as two separate numbers.

Why does long context use so much memory?

The cache grows with every token and has to stay in fast memory for the whole request. On a long context it can rival the model weights themselves, which is why long-context requests cost more and why shrinking the cache is an active research area.

Plain-language explainer

The KV cache, explained

What is the KV cache, and why does it matter for speed and cost?

When a model generates text, each new token needs to look back at every earlier token through attention. Recomputing that look-back from scratch for every token would be brutally slow. The KV cache stores each token's attention keys and values the first time they are computed, so every later step reuses them and only computes the newest token. The price is memory: the cache grows with context length, which is why long conversations eat GPU memory.

Last reviewed July 25, 2026

Reading is the slow way. Start with a free lesson you can operate right now.

Start free: Predict the next word →

Free, no code, no signup.

Then go deeper: Context window & KV cache Locked

What people get wrong

The KV cache is the model's memory of you. It is per-request working memory, thrown away after the response, and it never changes the model.
The cache stores your text. It stores attention keys and values, the vectors the model derived from your text.
A longer context window is just a bigger text box. Every token's keys and values must sit in GPU memory at once, so context length is a hardware budget.

Where you see it in real products

Very long chats slow down and hit limits because the cache keeps growing.
API providers sell cheaper cached input tokens: the keys and values for a repeated prompt prefix already exist.
Context-length specs on model cards are as much about GPU memory as about the model itself.

Common questions

What does the KV cache actually store?: The key and value vectors already computed for every token in the context. Without it, generating token 500 would mean reprocessing the previous 499 from scratch. With it, only the newest token is processed and the rest is read back.
Why does the first token take so much longer than the rest?: That is prefill: the whole prompt is processed at once to build the cache. Everything after it is a cheap single-token step reading from that cache. It is why providers quote time-to-first-token and tokens-per-second as two separate numbers.
Why does long context use so much memory?: The cache grows with every token and has to stay in fast memory for the whole request. On a long context it can rival the model weights themselves, which is why long-context requests cost more and why shrinking the cache is an active research area.

Related explainers

Context window Attention How LLMs work Prompt caching Speculative decoding

One idea at a time, in your inbox

New lessons and explainers, written the way these pages are. Now and then, not daily, and never a sales sequence.

We email you a confirmation link first. Unsubscribe in one click, any time. Privacy.

Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.