Chunk on document structure first, fixed size only as a fallback. Chunking decides what can ever be retrieved, so it caps the performance of every component downstream. It routinely moves retrieval quality more than switching embedding models does, and it is the cheapest thing in the pipeline to change.
A team swaps their embedding model, sees a two-point gain, and calls it a win. The same afternoon spent on chunk boundaries would likely have produced more, at no inference cost at all.
The reason chunking gets skipped is that it feels like plumbing. It is closer to schema design: a decision made once that constrains everything built on top of it.
At a glance
- A fact split across two chunks may be unretrievable by any model
- Structure-aware splitting beats fixed-size splitting on most real documents
- Overlap is a cheap insurance policy against bad boundaries
- What you embed and what you return to the model need not be the same text
- Tables and code are destroyed by naive character splitting
- Fix chunking before comparing embedding models, or the confound misleads you
Why does chunk size matter so much?
Because it sets two competing failure modes, and every strategy is a position between them.
Figure 1 — The two ways chunking fails
Chunks too small
A single fact gets split across a boundary. Neither half contains the complete answer, so neither ranks well for the query.
Pronouns lose their referents. “It requires approval” is meaningless without the sentence naming what “it” is.
Answers become unretrievable
Chunks too large
One embedding must represent many topics, so the vector drifts toward an average that matches nothing precisely.
Retrieval succeeds but returns mostly irrelevant text, diluting the context window and raising cost.
Precision collapses
The asymmetry matters: too-large chunks degrade quality gradually, while too-small chunks can make an answer impossible to retrieve at any k. When uncertain, err large.
The eight strategies
| Strategy | How it splits | Best for | Breaks on |
|---|---|---|---|
| Fixed character | Every N characters | Baseline only | Everything structured; splits mid-word and mid-table |
| Fixed token | Every N tokens | Predictable context budgeting | Semantic boundaries, same as above |
| Recursive separator | Tries paragraph, then sentence, then word | General prose; the sensible default | Documents with no consistent separators |
| Document structure | Headings, sections, list items | Markdown, HTML, technical docs | Unstructured or badly formatted sources |
| Semantic | Embeds sentences, cuts where similarity drops | Flowing narrative without headings | Cost; requires embedding during indexing |
| Sentence window | Embeds one sentence, returns neighbours | Precise retrieval with readable context | Extra storage and retrieval bookkeeping |
| Parent document | Embeds small chunks, returns the parent section | Best of both; strong general choice | Parent may exceed the context budget |
| Contextual prefix | Prepends a summary of the document to each chunk | Chunks that lose meaning in isolation | Indexing cost; needs a generation pass |
Parent document and sentence window share the insight that matters most here: the text you embed does not have to be the text you return. Small units retrieve precisely; large units read well. Decoupling them removes the central trade-off rather than balancing it.
What should you never split?
Four structures where naive splitting causes silent, total loss.
- Tables. A table severed from its header row becomes rows of unlabelled numbers. Keep the header with every fragment, or keep the table whole.
- Code blocks. Half a function is not retrievable and not useful when retrieved.
- Numbered procedures. Steps 4 through 7 without steps 1 through 3 will be followed anyway, which is worse than not retrieving them.
- Definition lists. A term separated from its definition destroys both.
Watch for this
Chunking failures are invisible in your index. Nothing errors, the vectors look fine, and counts appear reasonable. The only way to see the damage is to read a random sample of thirty chunks as plain text and ask whether each one still makes sense alone. Do this before building anything on top of the index.
How much overlap should you use?
Enough to survive a bad boundary, not so much that the index bloats.
Overlap works by ensuring a fact near a boundary appears complete in at least one chunk. The cost is duplicated storage and the risk of near-identical chunks competing in results.
Between 10 and 20 percent of chunk size is a reasonable starting band. Below that, boundary facts fall through; above it, you pay meaningfully more storage for diminishing protection.
Structure-aware strategies need less overlap, because their boundaries fall at natural seams rather than arbitrary offsets. That is a second reason to prefer them.
How do you benchmark this on your corpus?
Figure 2 — A one-day chunking benchmark
- STEP 1Reuse your eval setThe same labelled queries you used to pick an embedding model
- STEP 2Hold the model fixedChange only the chunker, or the result is uninterpretable
- STEP 3Index 3 strategiesRecursive as baseline, structure-aware, parent document
- STEP 4Score Recall@10Plus mean chunk size and total chunk count per strategy
- STEP 5Read the lossesInspect every query the winner failed; the pattern names your next fix
Step 5 is where the value is. Aggregate scores tell you which strategy won. The failed queries tell you why, and that usually points at a specific document type your chunker is mishandling.
Track chunk count alongside quality. A strategy that wins by producing four times as many chunks is also quadrupling your index cost, and that trade should be explicit.
Where do teams go wrong?
Using one strategy for every document type
A corpus mixing API reference, marketing pages and meeting transcripts does not have one right chunker. Route by document type and apply the appropriate strategy to each. This is unglamorous and reliably beats a single clever universal splitter.
Tuning chunking and the embedding model together
Two variables changed at once produce a number you cannot attribute. Since chunking often has the larger effect, the confound usually credits the model for the chunker’s work.
Ignoring what the chunk looks like to the model
A chunk beginning mid-sentence with an unresolved pronoun is confusing to a reader and equally confusing to an embedding model. Reading samples out loud catches problems that no metric surfaces.
What do experienced teams do differently?
They store rich metadata alongside every chunk and use it at query time.
Source document, section heading, position in document, and document type all cost nothing to store and enable filtered retrieval that pure vector similarity cannot express. Filtering to the right document type before ranking often beats improving the ranker.
They also prepend the section heading to the chunk text itself. A chunk beginning “Rate limits apply per organisation” retrieves far better when it opens with the heading path that gives it context, and the cost is a handful of tokens.
A short glossary
- Chunk
- A unit of text that is embedded and stored as a single vector, forming the smallest retrievable item.
- Overlap
- Text repeated between adjacent chunks so facts near a boundary appear complete in at least one of them.
- Recursive splitting
- Splitting on a prioritised list of separators, falling back to finer ones only when a chunk is still too large.
- Semantic chunking
- Placing boundaries where embedding similarity between consecutive sentences drops, rather than at fixed offsets.
- Parent document retrieval
- Embedding small chunks for precision while returning their larger containing section for readability.
- Contextual prefix
- A short document-level summary prepended to each chunk so it retains meaning in isolation.
Key takeaways
- Chunking caps retrieval quality, because a fact split across a boundary may be unretrievable at any k.
- Too-small chunks fail catastrophically; too-large chunks fail gradually. When unsure, err large.
- Structure-aware splitting beats fixed-size splitting on almost every real document type.
- The text you embed need not be the text you return, which is the insight behind parent document and sentence window retrieval.
- Tables, code blocks, numbered procedures and definition lists must never be split naively.
- Overlap of 10 to 20 percent is a reasonable starting band, and structure-aware strategies need less.
- Benchmark chunking with the embedding model held fixed, or the confound will credit the wrong component.
Frequently asked questions
What is the best chunk size for RAG?
There is no universal figure, because it depends on document type, embedding model sequence limits and how much context your generator can absorb. Rather than searching for a number, split on document structure and let sections determine size, using a maximum only as a safety cap.
How much overlap should chunks have?
Between 10 and 20 percent of chunk size is a sensible starting band. Overlap insures against facts falling across a boundary. Structure-aware strategies need less of it, because their boundaries land at natural seams rather than arbitrary character offsets.
Does chunking matter more than the embedding model?
Frequently yes, and it is far cheaper to change. Chunking determines what can be retrieved at all, so it caps every downstream component. Fix chunking first, then compare embedding models with the chunker held constant, or you cannot attribute the improvement.
How should I chunk tables and code?
Keep them intact wherever possible. A table severed from its header becomes unlabelled numbers, and half a function is neither retrievable nor useful. If a table must be split, repeat the header row in every fragment so each piece remains interpretable alone.
What is parent document retrieval?
A strategy that embeds small precise chunks for retrieval but returns the larger containing section to the generating model. It decouples the unit of matching from the unit of reading, giving precision in search and coherence in context without compromising either.
Is semantic chunking worth the extra cost?
Sometimes, mainly for flowing narrative text without reliable headings. It requires an embedding pass during indexing, which raises cost. For documents with clear structure, splitting on headings achieves similar boundaries for far less, so try structure-aware first.
Should I use the same chunking for every document type?
No. A corpus containing API reference, marketing pages and transcripts has no single correct chunker. Route by document type and apply an appropriate strategy to each. This unglamorous approach reliably beats searching for one universal splitter.
How do I know my chunking is bad?
Read thirty random chunks as plain text and ask whether each still makes sense alone. Chunking failures never raise errors, so inspection is the only detection method. Watch for fragments starting mid-sentence, unresolved pronouns and tables severed from their headers.
References
- Lewis, P., et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401.
- Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. MTEB: Massive Text Embedding Benchmark. arXiv:2210.07316. Context on why embedding performance varies by task and text type.
- Thakur, N., et al. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. arXiv:2104.08663.
