A large language model knows a great deal, and almost nothing about you. It has read the public internet and none of your contracts, your tickets, or your wiki. Two obvious fixes both break at production scale: retraining the model on your data is too slow and too expensive to keep current, and pasting all of your data into the prompt is too large to fit and too costly to send.
Retrieval was the pragmatic compromise. Keep the model general. At query time, find the few passages of your data that bear on the question, and hand only those to the model. That single idea — fetch the relevant pieces, then generate — is what RAG names. What has changed across three generations is not the idea. It is who decides what to fetch, how many times, and from where.
The idea never changed: fetch the relevant pieces, then generate. What changed is who decides.
Read the three generations as replacements and you will reach for the newest and skip the foundation. That is the wrong reading. Each generation wraps the one beneath it.
RAG 1.0: the deterministic pipeline
The first generation is a fixed assembly line. A document is split into chunks, each chunk is embedded into a vector, the vectors are stored in an index, the top few nearest to the query are retrieved, and the model writes an answer from them. Every stage is a specific line of code, and every run of the same input takes the same path.
The virtue here is also the ceiling: it is predictable. You can read the code and know exactly what will happen. Nothing improvises. The pipeline does the work; the model only writes the final paragraph.
In a 1.0 pipeline the code dictates the path. The model just writes the last paragraph.
But a fixed path has fixed failure modes, and they are not problems a bigger model fixes.
The first is single retrieval: the query is embedded once and searched once. If the wording was a poor match for how the answer happened to be phrased, there is no second pass to reframe it. The second is the query–document gap: a question and the statement that answers it live in overlapping but not identical regions of vector space, and a similarity search misses the ones that fall in the gap. The third is no reasoning across retrievals: a question that needs two facts stitched together from two places cannot be answered, because the pipeline retrieves once and never composes. The fourth is one corpus, one strategy: the pipeline is wired to a single index, cannot reach a database or an API or a chat thread, and applies the same retrieval strategy to every question regardless of what the question needs.
None of these are model-quality problems. They are consequences of letting code, not the model, decide the path.
The shift: who does the work
The second generation moves the decision. Instead of the code choosing the path and the model filling in the prose, the model chooses the path and the code becomes a box of tools it can call. Retrieval stops being a step in a line and becomes an action the model can decide to take — once, twice, or not at all.
RAG 1.0: the pipeline does the work and the model writes the answer. Agentic RAG: the model does the work and the pipeline holds the tools.
RAG 2.0: the agentic loop
Give the model a tool and a cycle and you get the second generation: think, act, observe, repeat. The model reasons about what it needs, calls a tool — a retrieval over the index, say — reads the result, and decides whether it has enough or should go again. It runs the loop a bounded number of times before it commits to an answer.
This directly answers two of the 1.0 limits. The single-shot problem dissolves, because the model can retrieve again with a reframed query after seeing what the first pass returned. And multi-step reasoning becomes possible, because the model can gather one fact, reason about it, and go back for the next.
A loop turns one blind retrieval into a conversation with the index.
The cost is latency, and it is the seed of the next generation’s central problem. A loop is sequential by nature: each pass waits for the one before it. Two or three passes is fine. A hard question that needs many passes, each waiting on the last, stacks into a delay no user will sit through.
RAG 3.0: orchestration
The third generation keeps the model in charge but changes three things at once: how it calls tools, how many sources it can reach, and who owns the retrieval machinery.
From sequential to parallel. Instead of looping one tool call at a time, the model fans out — issuing many independent calls at once and synthesizing the results as they return. Independent retrievals no longer wait in line; they run together and converge.
From one corpus to many. The 1.0 pipeline was wired to a single index. The orchestrator reaches across many heterogeneous sources — files, databases, code, analytics, mail, calendars, chat — and routes each question to the ones that can answer it.
From hand-tuned infrastructure to a managed service. Running your own vector store means owning chunking parameters, embedding calls, index tuning, and the monitoring around all of it — a standing maintenance burden that pulls effort away from the application itself. Vector search has largely become a commodity; a managed retrieval service absorbs the mechanics and frees the team to spend its effort on integration and business logic instead.
Which leaves the routing question — which tool answers which kind of question — and a quiet but consequential answer to it.
Routing as data, not code
In the first two generations the routing logic was buried in code: an if-this-call-that ladder that only an engineer could change and only a redeploy could ship. The third generation pulls that logic out of the code and writes it as plain instructions the model reads at runtime — a short Markdown file that says, in effect, “if the question mentions a meeting, check the calendar.”
The shift is larger than it looks. Routing becomes configuration rather than code: version-controlled, editable by someone who is not an engineer, and changeable without a deploy. The model reads the instructions the way a new hire reads an onboarding document.
Routing logic moved from code only an engineer can change to a document anyone can.
The generational matrix
Set the three generations side by side and the trajectory is one of steadily handing more judgment to the model while tightening the structure around it.
| Dimension | RAG 1.0 | RAG 2.0 | RAG 3.0 |
|---|---|---|---|
| Decision maker | Code, fixed | Model, sequential | Model, parallel |
| Data sources | One corpus | One corpus | Many, heterogeneous |
| Vector infrastructure | Self-hosted | Self-hosted | Managed service |
| Routing logic | Hardcoded | Implicit | Explicit, in files |
| Call pattern | One retrieve, one generate | A bounded sequential loop | Parallel fan-out, then synthesis |
The substrate
It is tempting to read this as each generation retiring the last. It does the opposite.
RAG 3.0 does not replace retrieval; it wraps it. Underneath the orchestration, the model is still calling a tool that chunks, embeds, searches, and returns the top passages — the exact machinery of 1.0. The loop of 2.0 is still running inside each branch of the fan-out. Every generation does the same fundamental thing; the newer ones just decide more about when and how to do it.
RAG 3.0 doesn’t replace the retrieval underneath it. It wraps it.
This is why the vocabulary of the first generation is not obsolete. Chunking, embeddings, top-k, hybrid search — these are the bedrock the whole structure stands on. An orchestration layer built on a shaky grasp of them is built on sand.
The far end of the trend is already visible: the boundary around “retrieval” dissolving until the model sits at the center like a kernel, with data sources — vectors, databases, APIs, message queues — attached as interchangeable peripherals over a common protocol. Whether or not it arrives in exactly that shape, the direction is clear. The model is becoming the thing that decides, and the sources are becoming things it talks to.
What to actually do
The lesson is an order of operations, not a leaderboard.
Build 1.0 first. Learn the substrate by hand — chunking, embeddings, hybrid search — on a single corpus. Do not skip this. It is the part everything else assumes you already understand.
Then build a 2.0 loop. One tool, one corpus. The point is to feel the conceptual shift: the model, not your code, deciding to call the retriever.
Scale to 3.0 only then. Orchestrate multiple tools and a managed service after the foundation is solid — not before. Without the foundation, orchestration looks like magic. With it, it is just good engineering.
Without the foundation, orchestration looks like magic. With it, it’s just good engineering.