What is KV cache?
A KV cache, short for key-value cache, is a memory optimization that lets a language model avoid recomputing the attention calculations for every token it has already processed, storing the intermediate key and value vectors from earlier steps so each new token only needs to compute attention against what’s stored rather than reprocessing the entire sequence from scratch. Without it, generating each new word in a response would require redoing the full attention computation over every word that came before, work that grows more expensive as a conversation gets longer; with it, that earlier work is computed once, kept in memory, and reused, which is what makes it practical for a model to generate long responses or hold long conversations at a usable speed.
Why generating text one token at a time creates a repeated-work problem
A transformer-based language model generates its output one token at a time, and each new token’s attention calculation needs to look back at every token that came before it in the sequence to decide what to attend to. Without any caching, producing the twentieth token in a response would mean recomputing the key and value vectors for all nineteen prior tokens all over again, even though those vectors haven’t changed at all since they were first computed. The same redundant computation happens for the twenty-first token, the twenty-second, and every one after that, with the wasted, repeated work growing larger with every additional token the model generates.
This is the specific inefficiency a KV cache exists to eliminate. Since the key and value vectors for a given token depend only on that token and the ones before it, not on anything generated afterward, those vectors never need to be recalculated once they’ve been computed the first time. Storing them and reusing them turns what would otherwise be a repeated, ever-growing computation into a much smaller amount of new work at each step, computing keys and values only for the newest token rather than the entire sequence again.
What actually gets stored, and why it grows the way it does
The cache holds two vectors, a key and a value, for every attention layer in the model and for every token that’s been processed so far. A model with many layers and a long context window ends up holding a correspondingly large number of these vectors, one pair per layer per token, which is why the KV cache’s memory footprint grows directly with both how long a conversation gets and how many layers the underlying model has. A short exchange with a small model keeps the cache modest, while a long, extended conversation with a large model can make the cache occupy a genuinely substantial share of the available memory, sometimes rivaling or exceeding the memory needed to hold the model’s own weights.
This growth pattern is exactly why KV cache memory management is one of the central engineering challenges covered in this collection’s broader discussion of LLM runtime. A runtime serving many simultaneous conversations needs memory for each conversation’s own growing cache, and if that memory isn’t managed carefully, a system can run out of capacity well before it runs out of raw computational power, forcing it to reject new requests or evict active conversations even though the actual processing capacity to serve them still exists.
How the cache changes what happens during the two phases of generation
Generating a response with a language model happens in two distinct phases, and the KV cache behaves differently in each. The first phase processes the user’s entire prompt at once, computing keys and values for every token in that prompt and populating the cache with all of them in a single pass, since the whole prompt is already available upfront. The second phase generates the response one token at a time, and at each step, the model only needs to compute new keys and values for the single newest token, checking those against everything already sitting in the cache from the prompt and every token generated so far.
This split explains why the first phase of generating a response often takes a noticeably different amount of time to produce its first visible output compared to how quickly each subsequent word appears, the first phase does a burst of parallel work across the entire prompt, while the second phase does a smaller, sequential amount of work at each individual step. Understanding this distinction matters for anyone trying to reason about where a model’s latency actually comes from, since the two phases have genuinely different performance characteristics and respond differently to changes in prompt length versus response length.
Why the cache is central to serving many users at once
A production system serving language model responses to many simultaneous users needs a separate KV cache for every active conversation, since each conversation’s history is different and its cached keys and values can’t be shared with anyone else’s. This connects directly to the batching mechanics covered in this collection’s discussion of inference serving, where a serving layer has to juggle multiple conversations’ worth of cache simultaneously while still finding ways to process requests together efficiently rather than treating each one in complete isolation.
The practical consequence is that a serving system’s total capacity is often limited by how much memory is available for all these simultaneous caches combined, not purely by how much raw computation the underlying hardware can perform. Two systems with identical processing power can serve very different numbers of simultaneous long conversations depending entirely on how efficiently each one manages its KV cache memory, which is why cache management sits at the center of nearly every serious discussion of how to serve language models at meaningful scale.
What happens when the cache runs out of room
Because KV cache memory is finite and grows with every token a conversation produces, a system eventually has to decide what to do when a conversation’s cache, or the combined cache across many simultaneous conversations, threatens to exceed the memory actually available. Some systems handle this by capping how long a single conversation’s context can grow before older tokens’ cached entries get evicted to make room for new ones, a decision that has direct consequences for how much of an older conversation a model can still meaningfully reference. Other systems manage capacity across many simultaneous conversations by deciding which ones to prioritize keeping fully cached and which to evict or pause when overall demand exceeds what the available memory can hold.
Getting this eviction and prioritization logic wrong produces exactly the kind of unpredictable, hard-to-diagnose degradation covered in this collection’s broader discussion of graceful degradation under load, a user whose conversation’s early cache entries get silently evicted may notice the model behaving as though it’s forgotten something it was told earlier, without any obvious signal that a memory management decision, rather than a genuine model limitation, caused that forgetting.
How techniques like quantization and shared caching push against the memory limit
Because KV cache memory pressure is such a persistent constraint, a considerable amount of engineering effort in this space goes into techniques that shrink the cache’s footprint without meaningfully hurting the model’s output quality. Storing the cached values in a lower-precision numerical format than the model’s full weights use is one common approach, trading a small, usually negligible amount of precision for a meaningfully smaller memory footprint per cached token. Another approach recognizes that when many users send requests sharing a common prefix, the same system prompt, the same set of instructions, the keys and values for that shared portion are identical across all of them and only need to be computed and stored once rather than separately for every single user.
These techniques matter because they directly expand how many simultaneous conversations a given amount of hardware can actually support, connecting back to the capacity and cost tradeoffs covered throughout this collection’s broader discussion of AI infrastructure scaling. A serving system that skips these optimizations pays a real, ongoing cost in reduced capacity for no corresponding benefit, since the optimizations exist specifically to reduce memory pressure without changing what a user actually experiences in the quality of their responses.
Why understanding the cache matters even for someone who never builds a runtime
Even a developer who never touches the internals of a model-serving system benefits from understanding what the KV cache actually does, because it explains several things that would otherwise seem like arbitrary platform behavior: why longer conversations or longer documents fed into a prompt cost more to process, why some platforms impose a maximum context length rather than allowing conversations to grow indefinitely, and why generation speed sometimes varies depending on how long a conversation has already become. All of these behaviors trace directly back to how KV cache memory accumulates and gets managed underneath, not to any arbitrary restriction imposed for its own sake.
This understanding also clarifies a subtlety in how caching costs actually work across a conversation, the tokens in an already-established conversation history don’t need their keys and values recomputed on a follow-up turn if a system is caching properly, which is a large part of why prompt caching features on production AI platforms exist and why they can meaningfully reduce the cost of a long, ongoing conversation compared to treating every single turn as an entirely fresh request that reprocesses the full history from the beginning.
Common mistakes teams make around KV cache
1. Ignoring KV cache memory as a distinct capacity constraint, planning hardware purely around raw computational throughput without accounting for how much memory simultaneous conversations will actually consume.
2. Setting no deliberate policy for what happens when cache memory runs out, allowing conversations to be silently truncated or evicted in ways that produce confusing, hard-to-diagnose behavior.
3. Skipping available memory optimizations, like shared caching across a common prompt prefix, that could meaningfully expand how many simultaneous users a given amount of hardware can support.
4. Assuming a model’s apparent “forgetting” of earlier context always reflects a genuine reasoning limitation, rather than considering that cache eviction under memory pressure may be the actual cause.
5. Treating every conversational turn as needing a full recomputation of the entire history, missing the cost and latency benefits that come from properly reusing an already-cached conversation prefix.
What connects these mistakes is failing to recognize that the KV cache is a first-class resource constraint in its own right, not an invisible implementation detail that takes care of itself, one that shapes how many users a system can serve, how long a conversation can meaningfully grow, and how quickly each new response actually arrives.
The deeper point about the KV cache is that it’s the specific mechanism that makes generating long, coherent text computationally practical at all, turning what would otherwise be an ever-growing amount of repeated work into a manageable, incremental cost at each step, and understanding how it behaves under memory pressure is what separates a system that scales gracefully to many long conversations from one that quietly degrades the moment real, sustained usage arrives.