What is modular RAG?

Summary slides

Why a monolithic RAG pipeline becomes genuinely difficult to improve over time

A RAG system built as one continuous, tightly coupled pipeline, where the chunking logic, embedding calls, retrieval, and generation are all interwoven directly rather than separated into distinct, independently addressable stages, tends to work fine initially but becomes considerably harder to improve as a team learns more about what actually needs fixing. Wanting to test a different embedding model, or add a reranking step, or swap in a different chunking strategy, each requires touching code that’s entangled with everything else around it, making even a focused, well-understood improvement risk introducing unintended side effects elsewhere in the pipeline.

This coupling problem compounds directly with the empirical, iterative improvement process covered throughout this collection’s discussion of AI native testing and RAG evaluation, improving a RAG system well requires the ability to change one component, measure the effect, and iterate, and a monolithic architecture makes this kind of focused, isolated experimentation considerably harder than it needs to be, every change carries the risk of unintended interaction with the rest of an entangled system.

How modular RAG actually separates a pipeline into independently addressable stages

Modular RAG organizes a RAG system around distinct stages with clear boundaries and well-defined interfaces between them, a query processing module handling reformulation and rewriting, covered throughout this collection’s discussion of query expansion and query rewriting, a retrieval module handling the actual search across one or more underlying mechanisms, covered throughout this collection’s discussion of vector search and hybrid retrieval, a post-retrieval module handling ranking, filtering, and reranking, covered in this collection’s discussion of reranking, and a generation module handling the final response synthesis. Each module takes a well-defined input and produces a well-defined output, and as long as that interface stays stable, the actual implementation behind any given module can change freely without requiring changes to the modules around it.

This clean separation is what makes the specific techniques covered throughout this collection genuinely composable rather than requiring a full pipeline rewrite every time one of them changes, swapping a chunking strategy affects only what happens before content enters the retrieval module, adding a reranking step slots into the post-retrieval module without touching query processing or generation at all, and each of these changes can be made, measured, and validated in relative isolation from the rest of the system.

Why modularity makes systematic evaluation and experimentation genuinely more practical

A modular architecture directly supports the kind of decomposed, stage-by-stage evaluation covered throughout this collection’s discussion of RAG evaluation, since each module’s contribution to overall quality can be measured somewhat independently rather than only as part of an undifferentiated end-to-end score. This makes it considerably more practical to run controlled experiments, testing two different retrieval strategies against the same query processing and generation modules, or testing two different rerankers against the same retrieval and generation modules, isolating the specific effect of the component actually being changed rather than confounding it with simultaneous changes elsewhere in the pipeline.

This experimental practicality compounds meaningfully over a RAG system’s ongoing development, a team working with a modular architecture can iterate on individual components with real confidence about what specifically drove an observed quality change, while a team working with a tightly coupled system has to work considerably harder to isolate which of several simultaneous changes actually caused a given improvement or regression.

Why modular RAG connects naturally to the hybrid strategies covered elsewhere in this collection

The hybrid RAG and hybrid search approaches covered elsewhere in this collection, combining multiple retrieval strategies or routing between them, become considerably easier to build well within a modular architecture specifically because the retrieval module’s clear interface already provides a natural place to plug in multiple underlying retrieval mechanisms and a routing or combination layer, rather than needing to bolt this complexity onto a pipeline that was never designed with multiple retrieval strategies in mind from the start. Modularity is less a competing idea to hybrid retrieval than an enabling foundation for building hybrid retrieval well.

This relationship is worth recognizing directly, the architectural patterns covered throughout this collection, hybrid retrieval, reranking, query rewriting, all become considerably more practical to build, combine, and maintain reliably when the underlying system is organized modularly from the start, rather than retrofitted onto an architecture that treated the entire pipeline as one continuous, undifferentiated process.

Why modularity introduces its own real cost that needs to be weighed against its benefits

Designing clean, well-defined interfaces between modules, and maintaining those interfaces as a system evolves, is itself genuine engineering work, and a small, simple RAG system with modest, unlikely-to-change requirements may not need this architectural investment at all, the flexibility modular design provides has to be weighed against the real upfront cost of designing and maintaining those clean module boundaries in the first place. This mirrors the same build-versus-simplicity tradeoff that shows up throughout this collection’s broader infrastructure discussions, more structure and flexibility isn’t free, and it’s worth investing in specifically when a system’s actual, anticipated need for iteration and experimentation justifies that upfront cost.

This is why modular RAG tends to matter most for systems expected to evolve considerably over time, ones where a team anticipates testing different embedding models, different retrieval strategies, and different reranking approaches repeatedly as the system matures, rather than for a narrow, stable system unlikely to need this kind of ongoing experimentation and component-level iteration.

Common mistakes teams make around modular RAG

1. Building a tightly coupled RAG pipeline for a system expected to evolve considerably, making later improvements considerably harder and riskier than a modular architecture would have allowed.

2. Over-investing in modular architecture for a small, stable system unlikely to need frequent component-level experimentation, adding unnecessary upfront complexity.

3. Designing module interfaces that leak implementation details, undermining the independence modularity is specifically meant to provide.

4. Evaluating a modular system only end-to-end, missing the stage-by-stage measurement modularity is specifically designed to make practical.

5. Treating modularity and hybrid retrieval as separate, unrelated concerns, missing how a modular architecture directly makes hybrid strategies considerably easier to build and maintain well.

What connects these mistakes is either underestimating or overestimating how much architectural investment a given RAG system’s actual, anticipated needs genuinely justify, modular design offers real, meaningful benefits for systems that will keep evolving, but it’s an investment that should be matched to a system’s actual trajectory rather than applied reflexively regardless of whether that flexibility will ever actually get used.

The deeper point about modular RAG is that a RAG system is rarely finished the moment it first works, the techniques covered throughout this collection, chunking, embedding models, reranking, query processing, all continue to improve over time, and a system architected to absorb those improvements incrementally, one well-isolated module at a time, stays considerably more maintainable and improvable over its working life than one that treats retrieval-augmented generation as a single, fixed pipeline built once and left largely unchanged.