Design a Document Processing Pipeline for LLMs and RAG

The question: design a pipeline that ingests and indexes large volumes of heterogeneous documents (PDFs, scans, Word files, slides, spreadsheets, HTML, emails) so they can be used by LLM applications such as RAG chatbots and extraction workflows. It must handle tens of millions of documents, keep up with updates and deletes, parse messy formats well, and do it at a reasonable cost. Most RAG quality problems start here, which is why interviewers ask it as a question on its own.

1. Clarify requirements

2. Architecture

3. Parsing heterogeneous documents

• Route by type and difficulty: cheap fast parsers first; escalate only hard pages (scans, complex tables) to OCR or vision models.
• Keep page numbers and bounding boxes so answers can cite the exact page.
• Store parsed output as structured intermediate files, so re-chunking doesn't require re-parsing.

4. Chunking, metadata and embedding

• Structural chunking by headings with the section path prepended; parent-child chunks for long sections. See RAG chunking strategies.
• Metadata per chunk: tenant, document id and version, source, ACL, page, section, language, timestamps, embedding model version.
• PII detection tags or redacts sensitive fields per tenant policy.
• Embedding on a GPU pool with large batches; content hashes skip unchanged chunks.

5. Throughput and cost

Backfill    : 50M docs × ~10 pages = 500M pages
Parsing     : fast path ~20 pages/s per CPU core → 25M core-seconds ≈ 290 core-days
              e.g. 1,000 cores → ~7 hours (plus OCR for the ~10% scanned pages,
              routed to a GPU/OCR pool with its own budget)
Chunks      : ~4 chunks/page → 2B chunks
Embedding   : 2B chunks at ~3,000 chunks/s per GPU → ~185 GPU-hours
              e.g. 40 GPUs → ~4.6 hours
Vectors     : 2B × 768 dims × 1 B (int8) ≈ 1.5 TB → sharded index per tenant group
Daily delta : 500K docs → ~1% of backfill work, runs continuously
(illustrative throughputs)

Backfills run at lower priority than live updates, in separate queues, so a new customer's 10-million-document import doesn't delay everyone's fresh edits.

6. Updates, deletes and consistency

• A document registry tracks every document's current version, status per stage, and checksums; it's the source of truth for reconciliation.
• Out-of-order events (an old update processed after a newer one) are rejected by version comparison.
• Deletes and permission revocations take a fast path: mark the document as hidden in the index immediately (a filter), then purge chunks and vectors asynchronously.
• Periodic reconciliation compares the registry with each source and with the indexes to catch missed events.

7. Reliability

• Each stage is idempotent (keyed by document id and version), so retries are safe; see idempotency.
• Poison documents (parser crashes, huge files) go to a dead-letter queue after a few attempts, with alerts.
• Per-tenant quotas and fair scheduling prevent one tenant from starving others; see multi-tenancy.
• Sandboxed parsers: documents are untrusted input (malformed files, macros, zip bombs).

8. Quality monitoring

• Parse quality signals: empty or garbled text ratio, OCR confidence, table detection rate per source and format.
• Freshness: lag from source change to searchable, per connector; delete propagation time.
• Downstream retrieval recall on a golden question set after any parser or chunker change; see LLM evaluation.