Context window & KV cache

What this lesson shows

Fixed context length; the KV cache reuses past keys/values for speed.

The question it opens with

Long chats get slow / the model forgets.

The walkthrough, in the lesson's own words

• Drag to a later word and watch the lit squares balloon. That growing triangle is the work.
• Flip the cache on. Watch the wasted work disappear.
• A tiny memory turns each new word's work from a steep curve into a gentle straight line, but it still climbs.
• Each lit square is one attention check: the new word paired with an earlier one.
• Drag all the way to the last word to feel how the work piles up.
• See how fast it fills? Word #6 checks 6 squares; word #1 checked just 1. Scale that up: word #1,000 does ~100× the work of word #10. A long chat keeps getting slower.
• To write the next word, attention re-reads every earlier word, the triangle from causal masking (lesson 3.4). So a long chat keeps getting slower.
• Per step the cost is roughly the position number, so word #1,000 costs ~100× word #10. Add every step together and the conversation's total work grows ~10,000× (the square), that's coming up next.
• On: a past word's Key and Value never change, so they're saved and reused. Only the new word adds one fresh row, the rest is remembered.
• Off: every step rebuilds the whole triangle from scratch. Slide to later words and watch the lit area balloon.
• Each cell pairs the new word with an earlier word using the keys and values from lesson 3.2. With the cache on, those keys and values are stored once and the new word only brings a fresh Query to compare against all of them, that's its single new row.
• Work for each NEW word: a steep curve without cache, a gentle straight line with it, far cheaper, but it still rises.
• The KV cache makes a long conversation basically free, like a normal cache that saves the answer.
• It only skips redoing old work; the model still re-reads the whole history every word, so cost keeps climbing as the chat grows.
• Without the cache, new work per step is re-read-everything; with it, just-the-new-word. That rising cost, plus the memory the cache itself eats, is a big reason vendors cap how much context they'll serve.
• The deeper reason: a model is only trained to handle up to some maximum length, and gets unreliable past it, the wall the next lessons run into.
• The cache eats serious GPU memory: every token stores keys and values for every layer. That memory bill is why modern models make groups of attention heads share one set of keys and values (GQA, grouped-query attention; MQA when all heads share one), shrinking the cache several-fold with barely any quality loss.
• It is also why providers sell "cached" input tokens at a steep discount: when your prompt's keys and values are already sitting in memory, they skip recomputing them. Lesson 6.4 turns that discount into a technique.
• A long chat with an assistant gets slower and pricier the further it goes, and eventually it forgets the start. Why?
• Every turn re-reads the whole context window and the KV cache grows with it, so cost and latency rise with conversation length. And the forgetting is the app's doing: when the window fills, the chat app trims or summarizes the oldest turns to stay under the limit (the bare model doesn't forget gracefully, it just errors past its maximum). The fix is managing context, trimming, summarizing, retrieving, not a bigger model.
• Attention grid: only the newest row is fresh work; earlier rows are remembered.
• Attention grid: the whole triangle is recomputed from scratch.
• Work per new word versus conversation length: a steep curve without cache, a gentle rising straight line with cache.

Key takeaway

Why long context is expensive, and why caching helps.

What you can do after this lesson

You can explain the KV cache and why long context is expensive.

Prefer reading first? This lesson has a companion explainer: KV cache

Where it leads: Attention is order-blind: how does it know word order?

This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.