RAG Chunking Strategies: Parsing, Chunk Size, Overlap and Metadata

Chunking strategies for RAG: document parsing, fixed vs recursive vs semantic chunking, chunk size and overlap, parent-child chunks, metadata and re-indexing.

Intermediate 3 min read · Updated 29 October 2026

In a RAG system, answers can only be as good as the passages retrieved, and passages can only be as good as the chunks you indexed. Chunking decides how documents are split before embedding: too large and a chunk mixes topics and wastes context; too small and it loses the surrounding meaning. This guide covers parsing, the main chunking strategies, chunk size and overlap, parent-child retrieval, metadata, and keeping the index fresh.

The ingestion pipeline

From raw files to searchable chunks. Most retrieval problems trace back to parsing or chunking, not to the embedding model.

Parsing comes first

PDFs lose structure: multi-column layouts interleave, headers and footers repeat on every page, tables become word soup. Use a layout-aware parser, and OCR for scanned pages.
Tables should be kept whole or converted to text rows with their headers repeated ("Plan: Pro, Price: $20, Seats: 5").
HTML and wikis: strip navigation and boilerplate; keep headings, lists and code blocks.
Images and diagrams: add captions or descriptions generated by a vision model if answers depend on them.
Keep the heading hierarchy: it's the most useful metadata you'll have.

Chunking strategies

Strategy	How it splits	Good for	Watch out
Fixed-size	Every N tokens, with overlap	Uniform text; a fast baseline	Cuts mid-sentence and mid-table
Recursive / structural	By headings, then paragraphs, then sentences, until under a size limit	Most documents; respects structure	Very uneven section lengths
Document-aware	By the format's units: functions in code, Q&A pairs in FAQs, clauses in contracts	Code, FAQs, legal, tickets	Needs a parser per format
Semantic	Where embedding similarity between adjacent sentences drops	Long unstructured text	Slower ingestion; boundaries can be unstable
Parent-child	Small chunks for matching, linked to a larger parent passed to the model	Precise matching with full context	More storage and logic
Proposition / summary	LLM rewrites content into self-contained statements or section summaries	Dense, reference-heavy material	Ingestion cost; risk of distortion

Chunk size and overlap

Two 500-token chunks with a 100-token (20%) overlap. Tokens 400-500 appear in both chunks, so a sentence that crosses the boundary is complete in at least one of them.

Common starting points are a few hundred tokens per chunk with 10-20% overlap; tune on your own evaluation set rather than trusting any default.
Smaller chunks give sharper matches but need more of them (and more context assembly) to answer broad questions.
Larger chunks carry more context but dilute the embedding and cost more tokens in the prompt.
Questions about specific facts favour small chunks; questions that need explanation favour larger ones or parent-child retrieval.

Parent-child (small-to-big) retrieval

Match on small child chunks for precision, then pass their larger parent section to the model so it sees the surrounding context.

Metadata on every chunk

{
  "chunk_id": "kb-4821#s3.c2",
  "doc_id": "kb-4821", "doc_version": 7, "parent_id": "kb-4821#s3",
  "title": "Refund policy", "section_path": "Billing > Refunds > Annual plans",
  "source": "help-center", "url": "https://help.example.com/refunds",
  "updated_at": "2026-10-02", "language": "en",
  "acl": ["group:all-customers"],              // who may retrieve this chunk
  "text": "Annual plans can be refunded within 30 days of ...",
  "embedding_model": "embed-v3"
}

Prepend context to the text you embed: document title and section path ("Refund policy > Annual plans: ...") so a chunk that says "within 30 days" is still findable.
ACLs let retrieval filter by the user's permissions before anything reaches the model; see RAG chatbot.
Dates and versions support freshness ranking and deletion.
Embedding model version tells you which chunks need re-embedding after an upgrade.

Keeping the index fresh

Incremental updates: on a document change, re-chunk only that document, upsert its chunks, and delete chunks from the old version (by doc_id and version).
Deletes must propagate: a removed or permission-revoked document must disappear from both indexes quickly.
Content hashing: skip re-embedding unchanged chunks.
Re-embedding on model change: build a new index in the background and switch atomically; see embeddings.

Measuring chunking quality

Build a set of questions with known supporting passages and measure retrieval recall@k for each chunking configuration, then end-to-end answer quality. Change one variable at a time. See LLM evaluation.

Where it’s used

Help centres

Structural chunking by heading, with titles prepended and short FAQ entries kept whole.

Code assistants

Chunks by function and class, with file path and imports as metadata.

Legal and policy

Clause-level chunks with parent sections, and strict version tracking.

In an interview

Mention parsing quality (tables, layout, OCR) before chunk size.
Pick a chunking strategy for the document types and justify the size.
Use parent-child retrieval when precise matching and broad context are both needed.
Store ACLs, versions and section paths on chunks, and handle updates and deletes.

Frequently asked questions

What is the best chunk size for RAG?

There is no universal best. A few hundred tokens with 10 to 20 percent overlap is a common starting point; tune it on your own questions by measuring retrieval recall, and use parent-child retrieval when you need both precise matches and broad context.

Which chunking strategy should I use?

Structural or recursive chunking by headings and paragraphs suits most documents; document-aware chunking suits code, FAQs and contracts; semantic chunking helps long unstructured text.