Design a RAG Chatbot for Company Documents (Enterprise AI Search)

The question: design a chatbot that answers employees' questions from company documents: wikis, policies, tickets, shared drives and chat archives. Answers must be accurate, cite their sources, respect who can see which document, and stay current as documents change. This is the most frequently asked AI engineering design question, and it tests RAG end to end: ingestion, retrieval, permissions, generation, evaluation and operations.

1. Clarify requirements

2. Define evaluation first

• A golden set of ~300 real questions with the documents that answer them and key facts, across departments, including questions with no answer in the corpus and questions about restricted documents.
• Metrics: retrieval recall@10, faithfulness (claims supported by cited chunks), correctness against key facts, refusal accuracy on unanswerable questions, and zero leaks on permission tests.
• See LLM evaluation.

3. Architecture

4. Ingestion

• Connectors per source pull content and permissions, using webhooks for change notifications plus periodic reconciliation crawls (webhooks get missed).
• Parsing and chunking: layout-aware parsing, structural chunks with title and section path prepended; see chunking strategies and document ingestion pipeline.
• Incremental updates: re-chunk only changed documents; upsert new chunks and delete old ones by document version.
• Queue-based workers so spikes (a whole drive shared at once) are absorbed; see message queues.

Corpus      : 5M docs × ~8 chunks = 40M chunks
Vectors     : 40M × 1,024 dims × 4 B ≈ 164 GB (float32) → ~41 GB as int8
Initial load: 40M chunks ÷ ~2,000 chunks/s embedding throughput ≈ 5.5 hours
Daily churn : ~1% of docs change → 400K chunks/day re-embedded, trivial
Keyword idx : BM25 over the same chunks in a search engine

5. Retrieval that respects permissions

The non-negotiable requirement. Each chunk stores the document's ACL (users and groups allowed). At query time, resolve the user's groups and apply them as a filter inside both searches, so restricted content never reaches the model or the prompt.

• Never retrieve first and filter afterwards in the prompt ("only use documents the user may see"): the model would see the restricted text.
• Group membership is cached with a short TTL; revocations from the identity provider invalidate it immediately.
• Permission changes in a source re-tag affected chunks quickly (they're metadata updates, no re-embedding).
• Very large ACLs (documents shared with thousands of groups) are expensive to filter; index by group, or partition the index by top-level group.

6. Retrieval and generation

• Rewrite follow-up questions into standalone queries using chat history.
• Hybrid search: BM25 for names, error codes and acronyms; vectors for paraphrases; fused with RRF; cross-encoder rerank to the best 6-8 chunks.
• Prompt: answer only from the provided sources, cite each claim with a source id, and say when the sources don't answer the question.
• Render citations as links to the source document (title, section, last updated).
• Optional grounding check on the answer before display for high-stakes areas (HR, legal).

7. Serving, latency and cost

Peak         : 10 questions/s
Latency plan : rewrite 300 ms (small model) | retrieval 80 ms | rerank 80 ms
               | first token ~700 ms → first text in ~1.2 s, then streaming
Tokens/query : ~600 system + 2,500 retrieved + 400 history + 300 output
Cost/day     : 100K × (3.5K in + 300 out) at illustrative mid-model prices
               ≈ 350M input + 30M output tokens/day → a few hundred dollars/day
Levers       : prefix caching for the system prompt, smaller model for rewriting,
               fewer but better chunks after reranking

8. Security

• Documents are untrusted: a page can contain injected instructions. The chatbot has no write tools, and output links are restricted to internal domains. See prompt injection.
• Logs and traces contain sensitive content: redact, restrict access, and apply retention.
• Audit who asked what and which documents were used, for compliance.

9. Operations

• Monitor ingestion lag per connector, failed parses, index freshness and permission-sync lag.
• Quality: thumbs-down rate, "I don't know" rate, sampled faithfulness scores; review negative feedback weekly and add cases to the golden set.
• Re-embedding on embedding model changes builds a new index in the background and switches atomically.
• See LLMOps.

Follow-up questions to expect

• *The answer is wrong but the document exists. Debug it.* Walk the stages: ingested? parsed? permission filter? chunk boundary? retrieval recall? reranked away? ignored by the model? See advanced RAG.
• *How do you answer "summarise all incidents this quarter"?* That's corpus-wide: use a structured query over ticket metadata or precomputed summaries, not top-k retrieval.
• *How do you stop answers from stale documents?* Prefer recent versions in ranking, show last-updated dates, and flag documents past a review date.