What is context compression?
Context compression reduces the size of any content headed into a model’s context, retrieved documents, tool outputs, reference material, to fit within an available budget while preserving as much useful information as possible, a broader technique than the memory compression covered elsewhere in this collection, which specifically addresses conversation history and stored memory. Context compression applies to any content category competing for space within the budgeting covered throughout this collection’s discussion of that topic, retrieved passages that are individually too long, verbose tool outputs carrying more detail than a task actually needs, reference documents exceeding what a budget can accommodate in full.
Why compression targets different content differently depending on what it actually is
Unlike memory compression’s narrower focus on summarizing conversational history into a shorter form, context compression has to handle genuinely varied kinds of content, a retrieved document, a tool’s raw output, a long reference passage, each with different structure and different signals for what actually matters versus what can be safely dropped. Compressing a retrieved document well means identifying and preserving the specific passages most relevant to the current query while dropping surrounding, less relevant material, compressing a verbose tool output means extracting the specific result a task needs while discarding formatting or metadata that doesn’t matter for the current purpose.
This variety is why context compression isn’t a single, uniform technique applied identically everywhere, it’s a family of related approaches, each tailored to the specific structure and purpose of whatever content is actually being compressed, connecting directly to the same content-type-aware handling covered throughout this collection’s discussion of chunking strategies, where different content types benefit from genuinely different processing approaches rather than one generic method applied indiscriminately across everything.
How extractive compression differs from generative summarization
Context compression can work by extraction, selecting and keeping the most relevant existing sentences or passages verbatim while discarding the rest, or by generation, using a language model to produce a new, shorter summary that captures the original content’s key points in different, more compact words. Extractive compression is cheaper and preserves exact original wording, which matters when precise phrasing carries meaning that a paraphrased summary might lose, while generative compression can achieve tighter compression and better readability but introduces the same interpretive risk covered throughout this collection’s discussion of memory compression, the compressing model’s own judgment about what matters could miss or misrepresent something the original content actually needed to convey.
Choosing between these approaches for a given content type is a real engineering decision, extractive compression tends to suit content where exact facts or figures matter, financial data, legal language, specific technical details, while generative compression tends to suit content where the overall gist matters more than exact wording, general background material, contextual explanation that doesn’t hinge on precise phrasing.
Why compression needs to happen at the right stage in the pipeline, not just anywhere convenient
Context compression can happen at different points, compressing a document once when it’s first ingested and stored, compressing retrieved content specifically at query time based on what a particular request actually needs, or compressing the final assembled context as a last resort if it still exceeds budget after everything else. Each point offers different tradeoffs, compressing once at ingestion is efficient since it only happens once regardless of how many future queries reuse that content, but it can’t tailor the compression to what any specific future query will actually need, while compressing at query time can target exactly what’s relevant to the current request but repeats that compression work on every single query that touches the same content.
This connects directly to the same pipeline design considerations covered throughout this collection’s discussion of embedding pipelines and RAG pipelines, deciding where compression happens is a genuine architectural choice with real cost and quality implications, not simply a detail to handle wherever happens to be most convenient within an existing system.
Why compression quality needs measurement against actual downstream task performance
The real test of whether context compression is working well isn’t whether the compressed content looks reasonable on casual inspection, it’s whether a model working from compressed content performs as well on its actual task as one working from the original, uncompressed version, connecting directly to the broader evaluation discipline covered throughout this collection’s discussion of AI native testing. Compression that looks fine to a human reviewer can still silently drop the specific detail a downstream task genuinely depended on, a failure that only becomes visible by measuring actual task performance rather than trusting a surface read of the compressed output.
This evaluation discipline matters because compression’s failure mode is exactly the quiet, hard-to-diagnose kind covered throughout this collection’s broader discussion of silent failures, a model working from compressed content that’s missing something important doesn’t produce an obvious error, it simply produces a subtly worse response, and only direct comparison against uncompressed performance reliably surfaces whether a specific compression approach is actually preserving what matters.
Why compression and selection work together rather than as substitutes for each other
Context compression and the context selection covered elsewhere in this collection address related but distinct problems, selection decides which candidates make it into context at all, while compression reduces the size of whatever candidates were already selected, and a well-designed system typically applies both together, selecting the most relevant candidates first, then compressing them further if the selected set still exceeds the available budget. Relying purely on compression without careful selection means compressing content that shouldn’t have been included in the first place, while relying purely on selection without compression means being forced to either exclude otherwise-relevant content entirely or exceed budget when even a well-selected set is individually too large.
Recognizing this complementary relationship matters for building a complete context assembly system, covered throughout this collection’s dedicated discussion of that topic, compression and selection both need to work correctly together, each addressing its own part of the same underlying budget-constraint problem this collection covers throughout its context management discussions.
Common mistakes teams make around context compression
1. Applying a single, uniform compression technique across genuinely different content types rather than tailoring the approach to each type’s actual structure and purpose.
2. Choosing generative summarization for content where exact facts and figures matter, introducing interpretive risk where extractive compression would have preserved precision better.
3. Compressing content at whichever pipeline stage happens to be most convenient rather than considering the real cost and quality tradeoffs of ingestion-time versus query-time compression.
4. Evaluating compression quality through casual inspection rather than measuring actual downstream task performance against the uncompressed original.
5. Relying on compression alone without proper selection, compressing content that should have been excluded from context entirely rather than included in a shrunken form.
What connects these mistakes is treating context compression as a simple, generic size-reduction step rather than a family of techniques requiring real judgment about content type, pipeline placement, and measured effectiveness, the specific approach that works well for one kind of content can perform poorly for another, and getting this right requires the same deliberate, content-aware engineering given to every other stage this collection covers.
The deeper point about context compression is that a fixed context budget and genuinely useful, complete information are often in direct tension, and resolving that tension well requires more than simply shrinking content until it fits, it requires understanding what each specific kind of content actually needs to preserve to remain useful, and applying compression deliberately, at the right stage, with the right technique, measured against real task performance, rather than treating compression as an unconditional, one-size-fits-all fix for anything too large to fit.