What is query rewriting?
Query rewriting transforms a user’s original request into a clearer, more self-contained version before it gets used for retrieval, specifically resolving references to earlier conversation turns, filling in context a standalone query would otherwise be missing, and correcting ambiguous phrasing, distinct from query expansion, covered in this collection’s dedicated article on that topic, which broadens a query with additional related terms or generates multiple search variations rather than resolving what a query means in context. Query rewriting solves a problem specific to conversational and multi-turn systems, a follow-up question like “what about the second one?” means nothing to a retrieval system without the conversation history that gives “the second one” an actual referent.
Why conversational queries are frequently incomplete on their own
In an ongoing conversation, people naturally refer back to things mentioned earlier rather than restating full context every single time, a user asking about “that pricing plan” after a previous exchange about specific plans, or “how does it compare to the other approach” without restating either approach’s name. This is normal, efficient human communication, but it creates a genuine problem for retrieval, a vector search or keyword search run directly against “how does it compare to the other approach” has no way to know what “it” and “the other approach” refer to, since that information exists only in the conversation history, not in the isolated query text itself.
Query rewriting exists specifically to solve this gap, using the conversation history to resolve these references before the query ever reaches the retrieval step, transforming “how does it compare to the other approach” into something like “how does the subscription pricing model compare to the pay-per-use pricing model,” a fully self-contained query that a retrieval system can act on meaningfully without needing any conversational context of its own.
How query rewriting resolves references using conversation history
A query rewriting step typically uses a language model, given the current query alongside the relevant preceding conversation turns, specifically instructed to produce a rewritten version that replaces ambiguous references, pronouns, shortened phrases, implicit comparisons, with the concrete, explicit information those references were pointing to. This is a comparatively narrow, well-defined task for a language model, unlike open-ended generation, it’s specifically reconstructing what a query means given available context, which tends to be a task language models handle reliably when given clear conversation history to work from.
This process needs to correctly identify which parts of the preceding conversation are relevant to resolving the current query’s references, and which aren’t, a long conversation history contains plenty of content that has nothing to do with the current query’s ambiguous references, and a rewriting step that indiscriminately incorporates irrelevant prior context risks producing an overly bloated or even incorrectly rewritten query rather than a cleanly resolved one.
Why unresolved references produce a specific, easy-to-miss retrieval failure
When a query containing unresolved references gets passed directly to retrieval without rewriting, the failure that results is often subtle rather than an obvious error, the retrieval system doesn’t crash or return an empty result, it simply searches for whatever literal words the ambiguous query actually contains, “the second one,” “that approach,” and returns whatever loosely matches those words, producing results that may look plausible on the surface while having nothing to do with what the user actually meant to ask about.
This connects directly to the broader discussion of silent, hard-to-diagnose failures covered throughout this collection, a system without query rewriting doesn’t fail loudly when a follow-up question’s references go unresolved, it fails quietly, returning confidently wrong or irrelevant results that look like a normal response rather than an obvious error, which is exactly the kind of failure mode that’s easy to miss during testing if conversational, multi-turn scenarios aren’t specifically included in evaluation.
Why query rewriting matters more as conversations get longer and more complex
A short, simple conversation with only a couple of turns rarely produces references complicated enough to genuinely confuse a retrieval system without rewriting, but as a conversation grows longer and covers more distinct topics, the number and complexity of references a follow-up question might contain grows correspondingly, a user several turns into a conversation might reference something from several exchanges earlier, using a shortened phrase that assumes the system remembers exactly what was discussed. This is why query rewriting’s value tends to increase with conversation length and complexity, a system handling only brief, simple exchanges may get away without rewriting more often than one handling long, involved conversations with extensive back-and-forth.
Recognizing this relationship matters for deciding how much investment query rewriting deserves for a given application, an application built around short, largely independent queries has less need for sophisticated rewriting than one built around extended, genuinely conversational interactions where later questions routinely depend on earlier context.
How query rewriting interacts with query expansion in a well-built pipeline
Query rewriting and query expansion address genuinely different problems and often get applied together in sequence within the same retrieval pipeline, covered throughout this collection’s broader discussion of that topic, rewriting first resolves a query’s references and ambiguity into a clear, self-contained, contextually complete version, and expansion then broadens that already-clarified query with related terms or additional phrasings to improve retrieval recall. Running these in the wrong order, or skipping rewriting entirely and expanding an ambiguous, unresolved query directly, tends to produce poor results, since expansion applied to an ambiguous query just produces several broadened variations of something that was never actually clear to begin with.
This ordering matters directly for building a well-functioning conversational retrieval pipeline, rewriting needs to happen first specifically because everything downstream, expansion, retrieval, ranking, depends on working with a query that actually reflects what the user meant, not the literal, possibly ambiguous text they happened to type in a given conversational turn.
Why evaluating query rewriting requires genuinely multi-turn test scenarios
Testing query rewriting well requires evaluation scenarios built around actual multi-turn conversations with genuine references and dependencies between turns, connecting to the broader discussion of AI native testing covered elsewhere in this collection, a test set consisting only of standalone, single-turn queries will never surface whether a rewriting step correctly resolves conversational references, since that capability simply never gets exercised by queries that don’t depend on any prior context at all.
This is why a team building conversational retrieval benefits from constructing test cases specifically designed around realistic conversational patterns, follow-up questions, shortened references, implicit comparisons across turns, rather than relying purely on single-turn test queries that would pass regardless of whether query rewriting is actually working correctly or not.
Common mistakes teams make around query rewriting
1. Passing conversational queries with unresolved references directly to retrieval, producing plausible-looking but irrelevant results that fail silently rather than obviously.
2. Applying query expansion to an ambiguous, unresolved query without rewriting it first, broadening something that was never clarified to begin with.
3. Incorporating irrelevant prior conversation turns into the rewriting process, producing an overly bloated or incorrectly resolved query.
4. Underinvesting in query rewriting for applications with genuinely long, complex conversational interactions where reference resolution matters considerably more.
5. Evaluating retrieval quality using only single-turn test queries, missing whether query rewriting actually handles the conversational references real usage will generate.
What connects these mistakes is underestimating how much conversational context shapes what a query actually means, a retrieval system built and tested only around standalone, fully explicit queries misses an entire category of real-world usage where later questions depend directly on what came before, and query rewriting is specifically the mechanism that bridges that gap before retrieval ever runs.
The deeper point about query rewriting is that a conversation’s later turns are rarely self-contained, they depend on shared context built up over the exchange, and retrieval systems built without accounting for this dependency will confidently search for the wrong thing whenever a user communicates the way people naturally do in an ongoing conversation, referring back rather than restating everything from scratch every single time.