Advanced RAG: Query Rewriting, Multi-Hop, Agentic RAG and GraphRAG

Advanced RAG techniques: query rewriting and decomposition, HyDE, multi-hop and agentic retrieval, GraphRAG, long context vs RAG, permission-aware retrieval.

Advanced 4 min read · Updated 29 October 2026

Basic RAG (embed the question, fetch the top chunks, generate) handles simple factual questions. It struggles with vague or conversational questions, questions that need facts from several documents, questions about relationships across a whole corpus, and corpora with strict permissions. Advanced RAG adds steps before retrieval (query transformation), during it (iterative and agentic retrieval, graphs) and after it (reranking, compression, verification). This guide covers the techniques and when each is worth its latency and cost.

Where basic RAG breaks

Failure	Example	Technique
Vague or conversational query	"what about for annual plans?" (depends on earlier turns)	Query rewriting with history
Vocabulary mismatch	User says "money back", docs say "refund"	Query expansion, HyDE, hybrid search
Multi-part question	"Compare the refund rules for monthly and annual plans in the EU"	Query decomposition
Multi-hop reasoning	"Who manages the team that owns the billing service?"	Iterative / agentic retrieval
Corpus-wide questions	"What are the main themes in customer complaints this quarter?"	Summaries, GraphRAG, map-reduce
Right document, wrong chunk	Answer split across a page break	Better chunking, parent-child retrieval
Too much irrelevant context	Top-10 chunks, 3 useful	Reranking, context compression

An advanced RAG pipeline

Pre-retrieval steps make the query searchable, retrieval may loop, and post-retrieval steps keep only what's useful. Each added step costs latency, so use them where the evaluation shows a gain.

Before retrieval: transforming the query

Conversational rewriting: turn "what about annual plans?" into "What is the refund policy for annual plans?" using the chat history. Almost always worth it in chat products.
Query expansion: add synonyms or several paraphrases, retrieve for each, and fuse the results.
HyDE (hypothetical document embeddings): ask the model to write a plausible answer, then embed that answer to search. It matches the style of documents better than a short question does. Risky when the hypothetical answer is confidently wrong.
Decomposition: split a compound question into sub-questions, retrieve for each, answer from the combined evidence.
Routing: classify the question to pick an index (policies vs API docs), a filter (product, region) or a non-RAG path (small talk, a SQL tool).

During retrieval: iterative and agentic RAG

In agentic RAG, retrieval is a tool the model can call repeatedly: it searches, reads, decides what's missing, and searches again, until it has enough or hits a budget. It handles multi-hop questions well, at the cost of several model calls and less predictable latency. Give it a step limit and require citations. See AI agent architecture.

Agentic RAG for a multi-hop question: each search is chosen based on what the previous one returned.

Corpus-wide questions: summaries and GraphRAG

Top-k retrieval can't answer "what are the main themes across 50,000 tickets?" because no chunk contains the answer. Options:

Hierarchical summaries: precompute summaries per document, per cluster and per collection; retrieve at the right level.
GraphRAG: use an LLM at ingestion to extract entities and relationships into a knowledge graph, cluster it into communities, summarise each community, and answer global questions from those summaries. Strong for themes and relationships, expensive to build and refresh.
Map-reduce: run the question over many chunks in parallel and combine the partial answers, for one-off analyses.
Structured data tools: if the answer is really a query ("how many tickets mention refunds?"), give the model a SQL or analytics tool instead.

After retrieval: reranking, compression, verification

Rerank with a cross-encoder and keep only the best few; see hybrid search and reranking.
Compress: extract only the relevant sentences from each chunk, cutting tokens and noise.
Order deliberately: put the strongest evidence first or last, since models attend less reliably to the middle of long contexts.
Verify grounding: check that each claim in the answer is supported by a cited chunk (an LLM or NLI check), and regenerate or answer "I don't know" when it isn't.

Long context vs RAG

	Put everything in a long context	Retrieve selectively (RAG)
Corpus size	Fits in the window (one contract, one codebase module)	Any size
Cost per query	High: every token, every time (prompt caching helps)	Low: only retrieved chunks
Latency	Slower prefill on long inputs	Retrieval adds tens of ms
Permissions and freshness	Hard to filter per user	Filter at retrieval time; index updates incrementally
Recall	Model may overlook facts mid-context	Bounded by retrieval recall

They combine well: retrieve generously, then use a long window to include more candidates than older systems could.

Debugging a wrong answer

Was the right document ingested, current and parsed correctly?
Did the permission filter exclude it?
Was the answer split across chunks?
Did the (rewritten) query retrieve it in the top 50? (retrieval recall)
Did reranking keep it in the final context?
Was it in the prompt but ignored or contradicted? (generation and grounding)

Log every step's inputs and outputs in a trace so this is a lookup, not guesswork; see LLMOps.

Where it’s used

Enterprise assistants

Conversational rewriting, permission-filtered hybrid search and grounding checks.

Research and analysis tools

Agentic multi-hop retrieval and map-reduce summaries over large collections.

Customer insight

GraphRAG or hierarchical summaries to answer theme-level questions over feedback.

In an interview

Name the failure first, then the technique that fixes it.
Use conversational query rewriting in any chat RAG design.
Propose agentic retrieval for multi-hop questions, with a step budget.
Debug wrong answers stage by stage, separating retrieval from generation.

Frequently asked questions

What is agentic RAG?

RAG where retrieval is a tool the model can call repeatedly: it searches, reads, decides what is missing and searches again until it has enough evidence or reaches a budget. It handles multi-hop questions at the cost of more calls.

What is GraphRAG?

An approach that uses an LLM to extract entities and relationships into a knowledge graph at ingestion, clusters and summarises it, and answers corpus-wide questions from those summaries.

Is long context replacing RAG?

Not for most systems. Long context helps when the material fits, but RAG remains cheaper per query, handles any corpus size, and filters by permissions and freshness. The two are often combined.