Document Chunking Strategies for RAG

Why Chunking Determines RAG Quality

Chunking is the most underrated lever in RAG. The chunks you index define what the retriever can return — not what's in the source document. A good question can fail to retrieve the right answer because the answer was split across two chunks, or buried in a chunk that's too large to score well against the query. Bad chunks cap the ceiling of every downstream optimization: better embeddings, better rerankers, and better prompts cannot recover information that was never in the candidate pool.

In our production RAG work, switching chunking strategy alone has moved Context Recall from ~60% to ~85% on the same corpus and embedding model. The gain is larger than most prompt-engineering changes and comes at one-time ingest cost rather than per-query latency.

Fixed-Size Chunking: The Default Baseline

Fixed-size chunking splits documents at a target token count (typically 256–512 tokens) with a configurable overlap (typically 10–20% of chunk size). It is the simplest strategy and the right default for prototyping.

Pros: trivial to implement, predictable index size, works acceptably on prose-heavy content (articles, reports, transcripts).

Cons: ignores semantic boundaries. A definition can be split from its example, a heading from its body, a table row from its header. This shows up as low Context Precision: the retriever returns the chunk containing the keyword but not the chunk with the actual answer.

For production systems beyond prototype, fixed-size is rarely the optimal choice — but it remains the right baseline to measure other strategies against.

Recursive and Semantic Chunking

Recursive chunking splits hierarchically: first by major separators (double newlines, section headers), then by minor separators (single newlines, sentences), only falling back to character splits if no semantic boundary fits within the size budget. LangChain's RecursiveCharacterTextSplitter is the most common implementation. This preserves paragraph and section integrity at low engineering cost.

Semantic chunking goes further: it embeds sentences, then groups adjacent sentences whose embeddings are similar (above a cosine threshold) into the same chunk. The implementation costs one embedding pass per sentence at ingest time but produces chunks that genuinely correspond to topic units. For Q&A over unstructured prose (interview transcripts, meeting notes, research reports), semantic chunking typically improves Context Recall by 10–20% over fixed-size.

The tradeoff is variable chunk size, which complicates retrieval-time budget management (top-K may pull very different token volumes). Combine semantic chunking with a hard upper bound (e.g., max 1024 tokens per chunk, splitting on sentence if exceeded).

Layout-Aware Chunking for Structured Documents

For PDFs, HTML, Markdown, or Word documents with explicit structure, layout-aware chunking dominates pure text-based approaches. Tools like Unstructured.io, LlamaParse, and Docling extract document structure (headings, tables, lists, figures) and let you chunk along those boundaries.

Effective patterns:

• Heading-aware chunks: never split across an H2 boundary. Each chunk includes its parent heading path as a prefix ("Product Manual > Installation > Network Setup"). The heading prefix gives the embedding model topical context that the chunk body alone may lack.
• Table preservation: keep tables intact in a single chunk where possible. If a table is too large, split by row groups, repeating the header row at each split.
• List preservation: never split an ordered list. Numbered steps lose meaning when "Step 3" appears in one chunk and "Step 4" in another.

Layout-aware chunking is what separates a RAG demo from a RAG product on enterprise documents. The engineering cost is real (parsing, structure extraction, edge cases) but the retrieval quality gain on structured content is the largest single improvement available.

Chunk Overlap and Context Windows

Overlap is the simplest defense against semantic-boundary splits. Standard guidance: 10–20% overlap (e.g., 50–100 tokens for a 512-token chunk). The reason is straightforward: if an answer happens to straddle two chunks, overlap ensures it appears intact in at least one of them.

Larger overlaps (above 25%) waste index space and inflate retrieval-time token consumption without proportional quality gains. Smaller overlaps (under 5%) provide little protection. The exception is sentence-window retrieval: index single sentences, then retrieve neighboring sentences (±N) at query time. This decouples retrieval granularity from the context window passed to the LLM.

Hierarchical and Parent-Child Chunking

Parent-child (sometimes called "small-to-big") chunking solves a fundamental tension: small chunks score better in retrieval (specific, focused), but small chunks lack the surrounding context an LLM needs to answer well.

The pattern:

1. Index small chunks (e.g., single paragraphs or sentences) for embedding retrieval
2. Maintain a pointer from each small chunk to a larger parent chunk (the section, page, or document)
3. Retrieve small but return parent for the LLM context

LlamaIndex's AutoMergingRetriever implements this. It is particularly effective for FAQ-style queries over long-form content (legal contracts, technical manuals) where the answer is a specific sentence but understanding it requires the surrounding context.

Measuring Chunking Quality

Do not pick a chunking strategy by intuition. Build a small evaluation set (50–200 questions with known-good answer documents) and measure Context Recall (does the retrieved set contain the answer?) and Context Precision (how much of the retrieved set is relevant?) for each candidate strategy. Ragas and TruLens both implement these metrics out of the box.

In our benchmarks across enterprise document corpora, layout-aware chunking with parent-child retrieval consistently outperforms fixed-size and pure semantic chunking on both Context Recall (typically 80–90%) and Context Precision (60–75%). But the magnitude of the gap depends entirely on document structure: on flat prose corpora, semantic chunking is competitive at lower engineering cost.

How Boolean & Beyond Approaches This

For most engagements, we typically run a 1–2 week chunking benchmark before locking in production architecture. We build a representative evaluation set from your actual user queries, implement 3–4 chunking strategies on a sample of your corpus, and measure Context Recall and Precision side by side. The output is a chunking strategy chosen by measured performance on your data, not by what worked on someone else's.

This evidence-based approach has saved clients from premature optimization (e.g., investing in layout parsing when their corpus was already flat prose) and from premature simplification (e.g., shipping fixed-size chunks on a corpus where heading-aware splits would have improved Recall by 20 points).

Summary: Chunking Implementation Priority Stack

1. Start with fixed-size (512 tokens, 15% overlap) as a baseline. Measure Context Recall and Precision against an evaluation set.
2. Move to recursive splitting before reaching for anything more complex — large quality gain at low engineering cost.
3. Add layout awareness if your corpus has meaningful structure (headings, tables, lists). This is the largest available single improvement on structured content.
4. Try parent-child retrieval if Context Precision is high but answer quality is low — it usually means chunks are too small to support good generation.
5. Reach for semantic chunking only when structure-based approaches do not apply (flat prose corpora) and when ingest cost is acceptable.
6. Re-evaluate after every embedding model or retriever change. Optimal chunking depends on the embedding model's effective context window and the retriever's ranking behavior.

The order matters. Most teams skip steps 1–4 and go straight to semantic chunking because it sounds sophisticated. In our experience, structure-aware chunking on enterprise content beats semantic chunking nearly every time at lower runtime cost.