The context window, explained

What is a context window, exactly?

The context window is the amount of text a model can consider at once, measured in tokens, the word fragments models actually read. Every request has a fixed maximum, and anything that does not fit simply does not exist for the model while it writes. It is the model's working memory: not what it knows from training, and not a saved transcript, just what is in front of it right now.

Everything shares that one budget. The hidden instructions the product adds, the whole conversation so far, documents you pasted, results from tools like web search, and the reply the model is writing all draw from the same pool. Input and output are one budget, which is why pasting a huge file can leave too little room for a long answer.

The stateless part surprises people. The model keeps nothing between requests. Each time you press send, the product re-sends the visible history and the model re-reads all of it from scratch before predicting the next tokens. What feels like memory is repetition. And when something no longer fits and gets dropped, it is not archived anywhere the model can reach. It is gone.

How big are context windows now?

Growth has been steep. GPT-2 shipped in 2019 with a window of 1,024 tokens. ChatGPT launched in late 2022 with about 4,000. In 2023 the GPT-4 family stretched from 8k to 128k, and through 2024 and 2025, 128k to 200k became the everyday standard while the first million-token models arrived. As of mid-2026, the flagship models from the major labs advertise windows around one million tokens, with reply length usually capped separately, often near 128k tokens.

To turn tokens into something you can feel, use the standard rule of thumb: a token averages about four characters of English, so 100 tokens is roughly 75 words. Code, rare words and most non-English languages cost more tokens per word, so the same window holds less of them.

What actually fills the window in a real product?

Far more than your message. In front of what you type, a chat product stacks a system prompt with instructions about tone, formatting, safety rules and the current date, plus your saved memory if the product has that feature. If the assistant can use tools, every tool's name, description and parameters are written into the window as text too. In a full-featured assistant, this fixed overhead alone can run to thousands of tokens before you say a word.

Then come the moving parts. The entire visible conversation is re-sent on every single turn. If the product does retrieval, known as RAG, the passages it fetched from your documents get pasted in. And when a model calls tools, every result, a web page, a file, a command's output, lands in the window as more text. In agent products, tool results are often the largest consumer of the window by far, dwarfing anything you typed.

That is why builders now talk about context engineering: trimming tool output, summarizing old turns, retrieving only the most relevant chunks. Every token of plumbing is a token not available for your problem, and since the whole window is re-processed each turn, waste also costs real money, softened in practice by prompt caching.

What happens when a conversation outgrows the window?

Something has to go. A raw API request that exceeds the limit is rejected with an error. Chat products avoid that by quietly making room: some keep a sliding window of only the most recent turns, others maintain a rolling summary that replaces old messages with a short recap. Either way the model answers as if the trimmed part never happened, which is exactly why a long chat remembers the last few messages perfectly and loses what you agreed on an hour ago.

Agents handle it deliberately and call it compaction. A coding agent that has spent an hour reading files and running commands will approach the ceiling mid-task, so the harness pauses, has the model write a dense summary of the session, keeps the recent steps verbatim, and continues from that. It works, but a summary of a summary blurs, so well-built agents also write important state to files that live outside the window entirely.

The other escape is to stop stuffing the window in the first place. Retrieval keeps your documents outside the model and pastes in only the passages relevant to the current question. And if you want to feel the mechanics, the interactive lesson on this site lets you watch a window fill up turn by turn and see exactly which messages fall out first.

Why isn't a bigger context window free?

First, compute. To write each new token, attention compares it against the tokens already in the window, so the work climbs steeply with length, quadratically in the basic transformer design. Serving a long context also means holding a per-token working state called the KV cache, which for large models can occupy tens of gigabytes of accelerator memory when the window is full. You feel this as latency, long prompts wait noticeably before the first word appears, and as price: you pay per input token, and as of mid-2026 some providers charge roughly double per token once a request crosses a threshold like 200k.

Second, reliability. The advertised window is a capacity claim, not a comprehension claim. Stanford researchers showed in 2023 that models use information at the start and end of a long context far better than information buried in the middle, with accuracy on the same question dropping by tens of points. The finding is known as lost in the middle. Later benchmarks such as RULER measured many models' effective length at a fraction of the advertised one, and a 2025 benchmark called NoLiMa found most tested models below half their short-context accuracy by 32k tokens once the question stopped sharing words with the text.

A July 2025 report from Chroma gave the everyday version a name: context rot. Across the 18 frontier models it tested, every single one grew less reliable as input length increased, even on deliberately simple tasks. A million-token window is real capacity. It is not a million tokens of full attention.

How do you work with the window instead of against it?

The practical rules fall straight out of the mechanics. A long chat does not only fail when it overflows. It degrades before that, because every stale turn is one more distraction competing for the model's attention and one more contribution to context rot.

• Start a new chat when you switch tasks. Dragging an unrelated hour of history into every request costs money, latency and accuracy, and buys you nothing.
• Ending a long session mid-project? Ask for a short handoff summary of decisions and open questions, then paste it into the fresh chat.
• Put what matters at the edges. Key instructions and the actual question land best at the start or the end of a long prompt, not buried in the middle.
• Paste the relevant section, not the whole document. Ten well-chosen paragraphs usually beat three hundred pages of maybe-relevant ones.
• In agent tools, watch the context meter and compact at a natural stopping point instead of letting it trigger mid-task.
• For facts the model should always know, use the product's memory or custom instructions rather than re-typing them into every conversation.

What people get wrong

Where you see it in real products

Common questions

Related explainers

More in Foundations