Retrieval-Augmented Generation (RAG) Architecture

Retrieval-augmented generation (RAG) answers a question by first retrieving relevant passages from your own documents and then asking a language model to generate an answer grounded in them. It lets a model use private, fresh and citable knowledge without retraining. Most RAG quality problems are retrieval problems, so the design effort goes into how documents are split, indexed, searched and ranked before the model sees anything.

The two pipelines

Indexing: chunking and embeddings

• Parse carefully: extract text with structure (headings, tables, lists) from PDFs, HTML and slides. Garbage parsing is a common hidden cause of bad answers.
• Chunk by structure, not just length: split at sections and paragraphs, typically a few hundred tokens, with some overlap. Too small loses context; too large dilutes relevance and wastes prompt space.
• Attach metadata: source, title, section path, date, owner and access permissions for every chunk, so retrieval can filter and the answer can cite.
• Embed each chunk with an embedding model and store vectors in a vector index; also index the text for keyword search.
• Keep the index fresh: re-index changed documents incrementally, delete removed ones, and record which embedding model version produced each vector (changing models means re-embedding everything).

Retrieval and re-ranking

• Hybrid search: combine vector similarity (good at paraphrases) with keyword search such as BM25 (good at exact names, codes and IDs), then merge the lists, for example with reciprocal rank fusion. See full-text search.
• Filter by permissions first: a user must never retrieve a chunk they couldn't open directly. Apply access filters inside the search, not after generation.
• Re-rank: a cross-encoder model scores each (question, chunk) pair more precisely than embeddings can, cutting dozens of candidates to the best handful.
• Query rewriting: expand abbreviations, resolve follow-up questions using chat history ("what about last year?"), or split complex questions into sub-queries.

Generation and grounding

Assemble a prompt with instructions, the retrieved passages (each labelled with its source), and the question. Instruct the model to answer only from the passages, cite them, and say when the answer isn't there. Treat retrieved text as data, not instructions, because documents can contain text that tries to redirect the model (indirect prompt injection).

"I couldn't find this in the documentation" is a good answer. Measure how often the system abstains correctly, not just how often it answers.

Evaluating a RAG system

Evaluate retrieval separately from generation. If the right passage wasn't retrieved, no prompt or model change will fix the answer.

Cost and latency

• Generation dominates both, so keep the context small: a few well-ranked chunks beat twenty mediocre ones.
• Cache embeddings of frequent queries and answers to repeated questions where data allows.
• Route easy questions to a smaller model and stream tokens to the user. See LLM serving.