04 Jan 2025 AI Engineering 5 min read

Agentic RAG: When Retrieval Loops Beat One-Shot Lookup

Classic RAG retrieves once and generates once. Agentic RAG lets the model decompose queries, retrieve iteratively, and critique its own context. We define the pattern, summarize the evidence from IRCoT, Self-RAG, and CRAG, state when a retrieval loop beats one-shot lookup, and show how to keep loop costs bounded.

The Limits of One-Shot Retrieval

Classic RAG is a fixed pipeline: embed the query, retrieve the top-k chunks, generate an answer. One retrieval, one generation, no second chance. This works for factoid questions against a clean corpus. It fails predictably on multi-hop questions, where the correct second search query depends on the answer to the first. The embedding of the full question rarely matches the document that holds the second hop.

It also fails silently. When the first retrieval returns noise, the generator still produces an answer — fluent, confident, and unsupported. A pipeline that cannot ask a follow-up question cannot recover from a bad first guess. That is the problem agentic RAG addresses.

Documents are chunked, embedded and indexed — vectors plus keywords. 1/4

What Makes RAG Agentic

Anthropic's engineering guide from 19 December 2024 draws a useful line: workflows follow predefined code paths, agents direct their own process. Agentic RAG applies this distinction to retrieval. The model decides whether to retrieve, what query to issue, whether the results are sufficient, and when to stop. Retrieval becomes a tool the model calls, not a stage it passes through.

The research is older than the label. ReAct (October 2022) interleaved reasoning steps with actions. IRCoT (December 2022) alternated chain-of-thought with retrieval. FLARE (May 2023) retrieved mid-generation whenever token confidence dropped. What changed by early 2025 is practicality: current models call tools reliably, so these loops now run in production code instead of research harnesses.

Query Decomposition in Practice

Query decomposition splits a compound question into sub-queries and retrieves for each. "Which of our two suppliers had the higher audit score in 2023?" becomes two lookups and a comparison. The numbers are concrete: IRCoT with GPT-3 (code-davinci-002) improved retrieval recall by 11 to 21 points and downstream QA by up to 15 F1 points on HotpotQA, 2WikiMultihopQA, MuSiQue, and IIRC, while cutting factual errors in reasoning chains by up to 50 percent.

Decomposition has a narrow scope. It helps when the question is compound or multi-hop. It does nothing when the corpus lacks the answer, and it adds one LLM call plus several retrievals of latency to every request — including the simple ones, unless a router filters those out first.

Iterative Search With Self-Critique

Self-critique closes the loop. Self-RAG (October 2023) trained models to emit reflection tokens: retrieve on demand, then grade each passage for relevance and each generated segment for support. Its 7B and 13B models outperformed ChatGPT on open-domain QA and fact verification. CRAG (January 2024) added a lightweight retrieval evaluator that scores retrieved documents and triggers corrective actions, including a web-search fallback, when confidence is low.

In production you rarely retrain a model for this. The practical translation is a graded loop: retrieve, let an LLM judge the chunks against the question, rewrite the query if they fail, retrieve again. Anthropic describes the same shape as the evaluator-optimizer workflow. Two to three iterations resolve most recoverable failures; more usually indicates a corpus problem.

When a Loop Beats a Single Lookup

A loop pays off under four conditions: the second search query depends on intermediate results; question vocabulary diverges from document vocabulary; evidence is spread across heterogeneous sources; a wrong answer is expensive. For factoid lookups against a well-indexed corpus, one-shot retrieval with a reranker is cheaper and faster. A three-iteration loop multiplies latency roughly threefold to tenfold.

A loop does not repair a weak index. If chunking, embeddings, or filters are broken, every iteration retrieves the same noise, and the critique step correctly rejects it forever. Fix single-shot retrieval quality first — hybrid search, reranking, better chunking. In our projects at Blue IT Systems, the loop is the last optimization we add, not the first.

The decision is measurable. Run your evaluation set through both paths and compare answer accuracy against tokens and latency per query. If the loop lifts accuracy by less than the reranker upgrade you have not shipped yet, ship the reranker first.

Cost Control in Retrieval Loops

Loops multiply token spend. Each iteration re-sends the accumulated context, so cost grows faster than linearly with iteration count. Without limits, a pathological query can cost fifty times the median. Cost control is therefore part of the architecture, not an afterthought.

The main levers are caps, model tiering, and caching. Grading and query rewriting do not need a frontier model: GPT-4o mini costs $0.15 per million input tokens and $0.60 per million output tokens (July 2024), against $2.50 and $10.00 for GPT-4o. Prompt caching cuts the price of re-sent context: Anthropic bills cache reads at 10 percent of the base input price, OpenAI discounts cached input by 50 percent.

Log token counts per query and per iteration from day one. A weekly percentile report shows which queries loop and why. In our experience, a small share of queries drives most loop cost; a router that sends easy questions down the one-shot path removes most of that spend without measurable quality loss.

Outlook for 2025

As of early January 2025, three developments seem likely to shape the year. First, reasoning models such as OpenAI's o1 (December 2024) will absorb part of the planning that loop code does today; the loop shrinks, it does not disappear. Second, frameworks such as LangGraph and LlamaIndex workflows are converging on graded retrieval loops as a standard component, which will make per-iteration telemetry the default rather than a custom build. Third, Anthropic's Model Context Protocol (November 2024) points toward standardized access to tools and data sources — exactly what retrieval loops need.

Our expectation: by the end of 2025, most production RAG systems will contain at least one model-controlled retrieval decision, while fully autonomous research agents remain the exception in regulated European environments. Cost per iteration will keep falling; the discipline of bounding iterations will not become optional. The teams that win will measure their loops as carefully as they built them.