KV cache: watch a long chat get slow, then fix it

1. 1Drag the slider to a later word. Each lit square is one attention check, and the lit triangle balloons as you go.
2. 2Flip the cache on. Past words keep their Key and Value, so only the newest row is fresh work.
3. 3Read the cost chart: a steep curve per word without the cache, a gentle straight line with it. Both still climb.

To write the next word, a language model runs attention over every earlier word. That is the lit triangle on the stage. Word six checks six squares, word one checked one. Scale that up and word one thousand does about a hundred times the work of word ten, and the conversation as a whole has done work that grows with the square of its length. This is why a long chat gets slower the further it goes.

The KV cache is the fix. A past word's Key and Value never change, so the model computes them once and keeps them. When the next word arrives it brings one fresh Query, compares it against the stored Keys, and adds one new row. With the cache on, the work per word goes from a steep curve to a gentle straight line. It still climbs, because the model still reads the whole history for every word, but it stops redoing what it already did.

The widget separates the small grid from the cost chart on purpose. The grid shows what one step looks like. The chart sweeps far past the grid so you can feel the difference between growing with the square and growing with a straight line.

The cache is the reason vendors sell cached input tokens at a steep discount: if your prompt's Keys and Values are already in memory, they skip recomputing them. It is also the reason context windows are capped. Every token stores Keys and Values for every layer, and that memory adds up fast on a GPU. Grouped-query attention, where several heads share one set of Keys and Values, exists to shrink this exact bill.

Paste this into any HTML page, course platform, wiki or slide tool that accepts an iframe. It runs with no account, sets no cookies of its own, and links back here.

Attribution is built in. If you write about it, a link to this page is all we ask.

Does the KV cache make long conversations free?

No. It only removes the wasted recomputation. Every new word still reads all the stored Keys and Values, so the per-word cost keeps rising with conversation length, and the cache itself keeps eating memory as it grows.

Why does my chat app forget the beginning of a long conversation?

Because the app trims or summarizes old turns to stay under the model's context limit. The bare model does not forget gracefully; it simply cannot run past its maximum length. Managing context is the app's job, not the model's.

Can I embed this KV cache widget in my own material?

Yes. Copy the iframe snippet on this page into a course page, a blog post or a slide. It runs without an account and links back to the full lesson on context windows and caching.