Plain-language explainer
Prompt caching, paying once for the part that never changes
What is prompt caching and how much does it actually save?
Most requests to an assistant repeat a long, identical opening: the system instructions, the tool definitions, the document you are asking about. Prompt caching stores the model's processed form of that prefix so the next request skips recomputing it. Providers charge a large discount on cached input and it comes back faster. The rules are strict: the prefix must match exactly from the first token, and the cache expires after minutes of disuse, so it rewards putting stable content first.
Last reviewed
Reading is the slow way. Start with a free lesson you can operate right now.
Start free: Predict the next word →Free, no code, no signup.
Then go deeper: Prompt caching: reuse the prefix Locked
What people get wrong
- It caches answers. It caches the processed prompt prefix, so the model still generates a fresh answer every time.
- Any repeated text is cached. The match has to start at the very first token, so one changed character near the top invalidates everything after it.
- It is free money. Some providers charge a premium to write to the cache, so a prefix reused only once or twice can cost more than not caching at all.
Where you see it in real products
- Coding agents that resend a long system prompt and the same files on every turn.
- Document assistants where the whole document is the shared prefix across many questions.
- Customer support bots with a large fixed instruction and knowledge block.
Common questions
- How should I order my prompt to get cache hits?
- Stable content first, variable content last. System instructions, tool definitions and long reference documents at the top; the user's question and anything with a timestamp at the bottom. Moving a single changing value to the top of a prompt is the most common reason caching stops working.
- How long does a cached prefix last?
- Minutes, not days, and it varies by provider. It is designed for a burst of related requests, such as one conversation or one agent run, rather than as long-term storage. Some providers offer a longer-lived tier at a different price.
- Is prompt caching the same as the KV cache?
- It is the same underlying idea reused across requests. The KV cache keeps computed keys and values inside a single generation; prompt caching persists that work for the shared prefix so a later request can start from it instead of from scratch.
Related explainers
More in Speed, cost and control
- What is model routing, and how do you choose which AI model to use?
- What is quantization, and how does it let big models run on small hardware?
- What does the temperature setting actually do to an AI model?
- Why are GPUs, and not CPUs, the hardware of the AI boom?
- How do reasoning models actually work?
- What is speculative decoding and why does it make models faster?
Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.