What is context window?

Summary slides

What a context window is

Every time a language model processes a request, it works with a single, bounded sequence of tokens — the context window is the maximum length that sequence is allowed to be, covering everything included in that one request at once: system-level instructions establishing how the model should behave, any content retrieved to help answer the question, prior turns in an ongoing conversation, and the model’s response as it’s being generated, since the output itself also consumes space within the same overall limit as it’s produced. This is a unified budget, not several separate allowances for each category — a request with a long system prompt and extensive retrieved content has correspondingly less room left for conversation history and for the response itself, because all of it draws from the same single, shared pool of available space.

It helps to think of the context window as the model’s entire field of view for a request — everything the model can reason about while producing its response has to fit inside it, and anything outside that window simply doesn’t exist from the model’s perspective, no matter how relevant it might have been. A model has no ability to reach outside its context window to check something it wasn’t; if a fact wasn’t included in the assembled context for a request, the model has no way to consult it directly during that request, and can only fall back on whatever it happens to remember from training, which — as covered elsewhere — is exactly the kind of fallback that produces plausible-sounding but potentially wrong answers rather than the grounded information the request needed.

Why length is measured in tokens, not words or characters

Context window limits are specified in tokens, and understanding why requires knowing what a token is: a chunk of text, generally smaller than a full word, that a model’s tokenizer breaks input into before processing it. Common English words are often a single token each, but longer or less common words frequently split into multiple tokens — a word like “internationalization” might break into several distinct pieces rather than counting as one token the way a simple word count would suggest. Punctuation, whitespace, and formatting characters typically consume tokens too, which means a request’s actual token count is usually higher than a naive word count would predict, sometimes substantially so for text with heavy formatting or unusual vocabulary.

This matters practically because a context window’s stated size — a number of tokens — doesn’t translate cleanly into an intuitive amount of text without running a tokenizer against the content in question. As a very rough approximation, a token is often something like three to four characters of typical English text, which puts a token limit at somewhere in the range of three-quarters of that same number in words — but this is just a rough approximation, and it varies meaningfully by language, by how technical or unusual the vocabulary is, and by how much of the content is code, structured data, or other non-prose text that tokenizes differently than ordinary written language does. Teams building anything where staying within a context budget matters need to count tokens directly, using the same tokenizer the target model uses, rather than estimating from word or character counts and hoping the approximation holds closely enough for their content and language.

Why the limit exists at all

Given how central the context window is to how a model operates, it’s worth understanding why a limit exists in the first place, rather than treating it as an arbitrary restriction that newer models simply haven’t gotten around to removing yet. The core reason is computational: the underlying attention mechanism that lets a model relate different parts of its input to each other has a computational cost that grows non-linearly with sequence length — roughly, doubling the length of the input more than doubles the compute required to process it, because every token’s attention calculation involves comparing it against every other token in the sequence. This means a context window twice as long isn’t merely twice as expensive to process — it’s meaningfully more expensive than that, which is a large part of why context window growth across model generations, while real and significant, hasn’t simply scaled up without any accompanying cost or engineering effort.

Memory is a second, related constraint: processing a long sequence requires holding a considerable amount of intermediate computation in memory simultaneously, and that memory requirement also grows with sequence length in a way that isn’t free to simply scale up indefinitely on the hardware available to run these models. Together, these two constraints — computational cost growing faster than linearly, and memory requirements growing alongside it — are the technical reasons context windows have a hard limit rather than being unbounded, and why meaningfully expanding that limit across model generations has required advances in model architecture and underlying hardware, not simply a policy decision to allow longer inputs.

How positional information keeps a long context coherent

Beyond the raw computational cost of attention, there’s a second, more subtle technical challenge in supporting long context windows well, worth understanding because it explains why simply making a model accept more tokens isn’t, by itself, sufficient to make it use those tokens well. A model needs to know not just what each token is, but where it sits relative to every other token in the sequence — whether a word appeared near the beginning of a long document or near the end of it, and how far apart two related pieces of information are. This positional information is what lets a model correctly track things like which pronoun refers to which earlier noun, or which instruction in a long system prompt applies to which later part of a request.

The technique used to encode this positional information has an effect on how well a model generalizes to context lengths it wasn’t originally trained on, and this is part of why simply retraining a model on longer sequences isn’t the only lever available for extending effective context length — architectural choices in how positions are encoded can meaningfully affect whether a model trained primarily on shorter sequences can still perform reasonably well when given a considerably longer one at inference time, or whether its performance degrades sharply the moment it’s pushed meaningfully past the lengths it saw most often during training. This is one of the less visible but consequential pieces of engineering behind the steady growth in usable context window sizes across model generations — it isn’t purely a matter of throwing more compute at the same underlying approach, but of architectural refinements that make longer sequences something a model can handle well, not merely accept without immediately erroring out.

Techniques that extend effective length beyond naive scaling

Given how steeply the raw computational cost of full attention grows with sequence length, model developers have adopted a range of techniques specifically aimed at extending how much context a model can practically handle without paying that full, naive cost at every step, and understanding the existence of these techniques — without needing to master their internals — helps explain why context window growth hasn’t simply tracked available compute in a straight line. Some approaches modify the attention mechanism itself so that each token doesn’t need to be compared against literally every other token in the full sequence, instead attending more selectively — to nearby tokens, or to a smaller set of specifically important ones — which reduces the computational cost at long lengths at some potential cost to how thoroughly distant parts of the context can directly inform each other.

Other approaches focus on training techniques that help a model generalize well to lengths beyond what it was primarily trained on, so that a model trained mostly on shorter sequences still performs reasonably when given something considerably longer at inference time, rather than degrading sharply the moment it’s pushed past its most common training length. Still others focus on system-level engineering — how the underlying computation is scheduled and executed on hardware — squeezing more effective context length out of the same underlying model architecture through smarter, more memory-efficient implementation rather than any change to the model’s parameters or training. None of this is something an application developer typically needs to implement directly, but understanding that it exists is useful context for interpreting why context window sizes have grown the way they have across model generations — not as a single, simple scaling knob being turned up uniformly over time, but as the product of several distinct kinds of engineering effort, each addressing a different piece of the same underlying cost and quality problem from a different angle.

What happens when a request exceeds the limit

Given that the limit is a hard technical constraint, it’s worth being concrete about what happens when an application tries to send more content than a model’s context window can hold, because the failure mode matters for how an application needs to be designed around it. Most model providers respond to an over-limit request with an explicit error, rejecting the request outright rather than silently processing a truncated version of it — which is the meaningfully safer failure mode from a correctness standpoint, since a rejected request is at least visibly, unambiguously a failure, rather than one that appears to succeed while quietly working from an incomplete, arbitrarily cut-off version of what was intended by the caller in the first place.

Some tooling and some application layers instead truncate content automatically before it ever reaches the model, cutting a request down to fit within the limit — usually from the end, though the specifics vary by implementation — and this is considerably more dangerous from a correctness standpoint precisely because it doesn’t fail visibly: a truncated request looks, from the outside, exactly like a normal, successful request, and the model dutifully answers based on whatever fraction of the intended content it received, with no indication to the end user that anything was cut off at all. This is exactly the kind of silent failure that connects back to the pipeline-level failure modes discussed in AI system reliability more broadly — a request that technically succeeded while working from incomplete information, indistinguishable from a fully correct response unless something in the surrounding system specifically checks for and flags truncation before it happens.

How context window sizes have grown, and what that growth solves

Context window sizes have grown substantially across successive model generations — early widely-used models supported a few thousand tokens, while more recent models support context windows that are, in some cases, well over a hundred times larger. This growth is significant and has opened up new use cases — processing an entire long document in one pass, holding a much longer conversation history without needing aggressive compression, including many more retrieved passages in a single RAG request than earlier, more constrained models ever could.

What this growth does not do, though, is eliminate the need for the deliberate context engineering discussed elsewhere: a larger window changes what’s technically possible to include, not what’s wise to include, and the lost-in-the-middle effect, cost scaling, and signal-dilution problems discussed there don’t disappear just because the underlying technical ceiling moved higher. A team with access to a much larger context window still benefits from selecting relevant content deliberately rather than including everything that now technically fits, for the same reasons that applied when windows were smaller — the reasons just apply at a larger absolute scale rather than being made irrelevant by that larger scale. Growing context windows are better understood as raising the point at which careful context management becomes strictly necessary rather than merely beneficial, not as removing the need for that management altogether.

The relationship between window size and cost

Because processing cost scales with how many tokens are included in a request, a larger available context window doesn’t mean using more of it is free — most providers charge based on the actual number of tokens processed in a request, for both the input and, often at a different rate, the output, which means a request that uses a large fraction of an available context window costs proportionally more than a shorter one, regardless of how generous the underlying model’s maximum window happens to be. This is a distinct consideration from the window limit itself: a team might have access to a context window large enough to include an entire large document in every request, but doing so on every single request, when only a small fraction of that document is relevant to a typical question, is an ongoing and often substantial cost that careful selection would avoid entirely.

Latency scales with token count too, in both directions — a longer input takes measurably longer to process before the model can begin generating a response, and a longer generated output takes longer to produce and stream back. For an interactive application where response time matters to the person waiting for it, this latency cost is often a more immediately felt constraint than the pure dollar cost of extra tokens, and it’s a practical reason to keep context tight and relevant even in situations where raw dollar cost alone might otherwise seem to justify a more generous, less disciplined approach to what gets included in a request. A team optimizing purely for cost per token might reasonably conclude that a slightly larger context is affordable; a team also accounting for how that larger context affects the time a person spends waiting for a response often reaches a meaningfully different, more conservative conclusion about what’s worth including.

Nominal size versus effective, reliably usable size

A model’s advertised context window size — the maximum number of tokens it technically accepts — is not the same thing as the amount of content it can reliably reason well over, and this distinction matters enough to name directly rather than assume the two are interchangeable. The lost-in-the-middle effect discussed in the context engineering discussion means that content positioned in the middle of a long context is measurably less reliably attended to than content near the beginning or end, which means a model’s practical, effective capacity for reliable use is often meaningfully smaller than its nominal maximum, particularly as that maximum grows very large.

This gap between nominal and effective capacity is worth testing directly for any application where it matters, rather than assumed from a model’s advertised specifications alone — a team can construct a test that places a checkable fact at different positions within contexts of varying length and measures how reliably the model retrieves it, which gives a much more grounded, empirical sense of a model’s reliable effective context length than the advertised maximum alone would suggest. Designing an application around a model’s nominal context limit, without accounting for this gap, is a common and avoidable way to end up with a system that technically stays comfortably within its stated limits while still quietly underperforming precisely on the requests that push toward the upper end of what that nominal limit allows — the exact cases, in other words, where a team is most likely to assume the system is working well simply because nothing technically errored out along the way.

Multi-turn conversations and the compounding budget problem

The relationship between context window size and cost, discussed above for a single request, becomes a compounding concern once a conversation extends across many turns, because a naive implementation resends the entire prior conversation history with every single new message, which means the token cost of each successive turn grows as the conversation itself grows — the tenth message in a conversation costs meaningfully more to process than the first, purely because it’s now carrying the weight of nine prior turns along with it, even though the actual new content in that tenth message might be no longer than any of the earlier ones.

This compounding effect is what makes the compression techniques discussed in context engineering — summarizing older turns while keeping recent ones verbatim — a practical necessity rather than an optional refinement for any application expecting conversations to run long, since without it, a sufficiently long conversation will eventually consume an outsized share of the available context window purely on replaying its history, leaving correspondingly less room for the retrieved content or fresh information that a later turn in that same conversation might need. It’s also an ongoing cost consideration distinct from the pure context-window-limit question: even a conversation that technically stays within the window’s hard limit throughout its entire length can become considerably more expensive to continue, turn over turn, than a team accounted for when estimating costs based only on a single, isolated request rather than a full, realistic multi-turn exchange.

Handling documents longer than a single context window

Even with the substantial growth in context window sizes across model generations source documents — a lengthy legal contract, a large codebase, an extensive research report — can still exceed what a single request can hold, which raises a practical question distinct from everything covered so far: how does an application process content too long to fit in one context window at all, rather than merely too long to include casually alongside other content.

A few established patterns address this. Chunking, already discussed at length in the context of retrieval, is the most common: breaking a long document into smaller pieces at ingestion time, so that only the relevant pieces need to be included in any request, rather than the document ever needing to be processed as one single unit at all. Map-reduce-style summarization is a different pattern, suited to tasks that need to process an entire long document rather than just a relevant excerpt of it — the document is split into sections small enough to fit individually within the context window, each section is processed or summarized independently, and those independent results are then combined, possibly through a further summarization pass, into a single final output that reflects the document as a whole despite no single request ever having held the entire thing at once. Sliding-window processing, a related pattern, moves a fixed-size window progressively through a long document, with some overlap between consecutive windows to avoid losing information that happens to sit right at a boundary between them, useful specifically for tasks like extracting information that might appear at any point in a long document without requiring the document to be intelligently pre-chunked by relevance ahead of time. Choosing between these patterns depends on the actual task: retrieval-style chunking suits question-answering over a large corpus well, while map-reduce summarization suits tasks that need a faithful understanding of an entire document as a whole, and conflating the two — using retrieval-style chunking for a task that needed a full-document understanding, for instance — is a common, avoidable source of an application quietly missing information that technically existed in the source material but never made it into any single request in a form the model could reason over completely.

Context window’s relationship to memory and retrieval

Given that the context window is a hard, per-request limit, it’s worth being explicit about how it relates to the broader concepts of memory and retrieval discussed elsewhere, since the three are easy to conflate but serve different roles. The context window is the mechanism — the actual technical space available in a request. Retrieval is a strategy for deciding what to put into that space when working with a knowledge base larger than could ever fit within it directly. Memory is a strategy for deciding what to carry forward across separate requests and sessions, given that nothing persists automatically in the model itself between one context window and the next.

All three concepts exist because of the same underlying fact: the context window is finite, and a useful AI application almost always needs to work with more total information — a larger knowledge base, a longer conversation history, more accumulated memory — than any single context window could ever hold directly. Retrieval and memory are, in that sense, both answers to the same basic constraint the context window imposes, just addressing it from different angles: retrieval selects from a large, mostly-external body of knowledge, and memory selects from a smaller, curated, explicitly maintained store of what’s been deliberately deemed worth carrying forward, but both exist specifically because the context window itself can never simply hold everything that might conceivably be relevant.

It’s worth being precise about one further distinction here, since it trips up teams new to building these systems: a larger context window doesn’t reduce the need for retrieval or memory the way it might seem to at first glance — it changes the scale at which those strategies operate, not whether they’re needed at all. A knowledge base of a few dozen short documents might, with a sufficiently generous context window fit in its entirety within a single request, making retrieval in the strict sense almost unnecessary for that small-scale case. A knowledge base of many thousands of documents, or one that grows continuously over time, will never fit within any context window, however large that window eventually becomes, because the knowledge base’s growth will simply continue to outpace whatever the current generation of models happens to support. For any application operating at that second, more realistic scale, retrieval and memory remain necessary regardless of how much larger context windows get in future model generations — the finite window is a permanent architectural fact for these systems, not a temporary limitation on the verge of being fully engineered away.

A worked example: budgeting a context window in practice

These concepts are easier to see clearly against a concrete case, so consider a support assistant working with a model that has a context window of a moderate, realistic size for this kind of application. The system instructions establishing the assistant’s role and behavior consume a modest, fixed amount of that budget on every single request, regardless of what the question happens to be. Retrieved documentation, selected deliberately rather than included in bulk, consumes a variable amount depending on how many passages the retrieval and reranking steps determined were relevant to the current question — a well-tuned system keeps this focused rather than padding it out simply because more room happens to be technically available.

Conversation history, compressed as discussed in the context engineering practice — recent turns kept in full, older ones summarized — consumes another variable portion, growing modestly as a conversation lengthens rather than growing without bound the way an uncompressed raw transcript eventually would. And space needs to be explicitly reserved for the model’s response, since the output shares the same overall budget as everything sent in — a request that fills the input side of the context window right up to its technical limit leaves the model with no room to produce an answer at all, an avoidable design mistake that shows up more often than it should in systems that don’t explicitly account for the output’s share of the shared budget when assembling the rest of the request.

Working through the actual numbers makes the trade-offs concrete. Suppose the assistant’s system instructions consume a modest, fixed few hundred tokens on every request. A well-tuned retrieval step, returning perhaps three to five relevant passages rather than the dozens a looser system might return, might consume a low thousands of tokens — a cost, but a bounded and predictable one, distinct from what an unfiltered, dump-everything-that-might-be-relevant approach would cost instead. Compressed conversation history, growing modestly rather than linearly with the raw length of the exchange, adds a further, comparatively small amount. Reserving headroom for the model’s response — enough for a complete, thorough answer rather than a truncated one — rounds out the budget. Add these together against the model’s actual available window, and what’s left as spare capacity is the team’s quantified safety margin: room to occasionally retrieve a few more passages for an unusually complex question, room for a conversation to run a little longer than typical before compression kicks in more aggressively, without ever risking the hard limit on an ordinary request. A team that never does this kind of concrete accounting is flying blind on exactly the numbers that determine whether their system will behave predictably at the edges of its normal usage range or whether it will start failing, confusingly, the first time a slightly-longer-than-typical request comes in.

Common misunderstandings about context windows

A consistent set of misunderstandings shows up across teams working with these systems, worth naming directly because each one leads to an avoidable problem. The most common is estimating token count from word or character count rather than running the target tokenizer against content, which produces budget estimates that can be meaningfully off, particularly for text that’s heavy on code, unusual vocabulary, or non-English language, where the rough word-to-token approximation breaks down considerably more than it does for typical English prose.

A second is treating a model’s advertised maximum context window as equivalent to its effective, reliably usable length, building a system that technically stays within the stated limit while still suffering from the lost-in-the-middle degradation that shows up well before that nominal maximum is reached. A third is forgetting that the model’s output shares the same budget as everything sent in, assembling an input that leaves too little room for a complete response and then being confused when responses come back truncated or cut off mid-thought for reasons that have nothing to do with the model’s actual reasoning quality. A fourth is assuming a larger context window in a newer model eliminates the need for careful context engineering, when in fact it primarily just raises the scale at which that same discipline remains necessary. And a fifth is failing to handle the overflow case at all — never checking token counts before sending a request, and only discovering a hard limit exists the first time an unusually long request in production triggers a rejected call or, worse, a silently truncated one, rather than having designed for that boundary condition deliberately and tested against it well before it ever showed up unplanned in front of a user.

What unites every one of these misunderstandings, and every technical detail covered before them, is a single underlying point worth carrying forward: the context window is a hard, physical constraint on what a model can process in any single request, not an abstract inconvenience that better models eventually make irrelevant. It has grown substantially and will likely keep growing, but growth in the nominal maximum has never been the same thing as growth in what’s wise or reliable to include, and treating the two as equivalent is the single most consistent source of the avoidable problems covered throughout this page — from silent truncation to compounding multi-turn cost to a false sense of safety about how reliably a model uses everything technically present in a very long context.

Teams that internalize this distinction early tend to build noticeably more robust systems than teams that only discover it after something breaks in production, because every one of the practices that follows from it — measuring tokens with the actual tokenizer rather than a rough approximation, retrieving narrowly rather than broadly, compressing conversation history deliberately rather than letting it grow unchecked, reserving headroom for the model’s response, and treating a model’s nominal maximum as an upper bound rather than a working target — is far easier to build in from the start than to retrofit onto a system that was designed without them and is already serving traffic. Retrofitting means finding every place a raw document gets concatenated into a prompt without a token check, every place conversation history accumulates without a compression step, and every place a team quietly assumed the nominal maximum was the number that mattered; building it in from the start means those decisions are made once, deliberately, before the system depends on them being right. Understanding the context window precisely — what it measures, why it’s limited, what happens at its edges, and how its nominal size differs from its effective one — is foundational to building AI systems that behave predictably rather than ones that merely happen to work until a request finally pushes hard enough against a boundary nobody had designed around.