RAG Chunking Strategies: How to Split Documents for Retrieval
Chunking is the highest-leverage, least glamorous decision in a RAG system. Get chunk boundaries wrong and no amount of reranking or prompt engineering recovers the information that got split across two chunks or buried inside one that is too large to embed precisely. There are four families of chunking strategy in production use in 2026: fixed-size chunking with overlap, semantic chunking that splits on meaning boundaries, structural chunking that respects document shape, and late chunking, a newer technique that embeds the full document before pooling token representations into chunks. None of them is universally correct. The right choice depends on how homogeneous your source documents are, how much compute you can spend at ingestion time, and whether your evaluation harness can actually detect the difference.
Fixed-Size Chunking: The Baseline Everyone Starts With
Fixed-size chunking splits text into windows of a set token or character count, typically 256 to 512 tokens, with a 10 to 20 percent overlap so a sentence spanning a boundary is not lost entirely from either chunk. It is cheap, deterministic, and easy to reason about, and it works well on homogeneous prose like policy documents, articles, and most support documentation. Its weakness is that it is indifferent to meaning: a fixed 400-token window will cut a table in half, split a bulleted procedure across two chunks, and occasionally end mid-sentence in a way that damages embedding quality. Start here for any new corpus, measure recall@k on your golden set, and only add complexity if the numbers justify it.
- Typical range: 256 to 512 tokens per chunk with 10 to 20 percent overlap
- Cheapest strategy to implement and the easiest to reason about at scale
- Blind to document structure, so tables and procedures get split arbitrarily
Semantic Chunking: Splitting on Meaning Boundaries
Semantic chunking embeds sentences or small groups of sentences, measures the similarity between consecutive segments, and inserts a chunk boundary where similarity drops sharply, on the theory that a topic shift is a natural place to split. It genuinely helps on long-form, mixed-topic documents like meeting transcripts, multi-section reports, or long email threads where a fixed window would otherwise mash two unrelated topics into one chunk. The cost is real: it requires an embedding pass just to determine boundaries, adding ingestion-time compute and latency, and the boundary-detection threshold needs tuning per corpus or it either over-splits into tiny fragments or under-splits into oversized chunks. Reserve it for corpora where topic drift within a document is common and measurably hurting recall today.
Structural Chunking: Respecting Document Shape
Structural chunking splits along the document's own structure: markdown headers, table boundaries, code blocks, or, for engineering and ERP documentation, sections like a title block, bill of materials, or revision history. This is usually the best default for technical corpora because the document author already encoded meaningful boundaries, and ignoring them is what causes a 400-token fixed window to cut a specification table in half. Implementing it requires a parser that understands the source format, whether that is markdown, structured PDF, or an exported ERP document, and each chunk should carry structural metadata such as section heading or table name so retrieval can filter and the generator can cite precisely.
- Split on headers, tables, and logical sections rather than a fixed token count
- Attach structural metadata, section heading, table name, or drawing sheet, to every chunk
- Strong default for ERP documentation, specifications, and other formally structured content
Late Chunking: Embedding the Full Document First
Late chunking, introduced widely in 2024 and now common in production embedding models with long context windows, inverts the usual order: embed the entire document (or a large section) as one pass through a long-context embedding model, then pool the resulting token-level embeddings into chunk-sized vectors after the fact. Because every chunk's vector was computed with the full document in context, references and pronouns that would be ambiguous in an isolated 400-token window get resolved correctly, which measurably improves recall on documents with heavy cross-referencing. The tradeoff is that it requires an embedding model with a long enough context window to hold the full document, and re-embedding is more expensive when a document is updated, since you cannot re-embed a single chunk in isolation.
Choosing Chunk Size and Overlap in Practice
Do not pick chunk size from a blog post benchmark; pick it from your own recall@k measurements. Start with 400 tokens and 15 percent overlap as a baseline, run your golden evaluation set, then sweep chunk size up and down while holding everything else constant. Smaller chunks generally improve retrieval precision but hurt generation because the model receives less surrounding context per chunk; larger chunks do the reverse. The right answer is usually different for a FAQ corpus than for a set of engineering specifications, which is why structural or hybrid strategies that vary chunk size by document type tend to outperform any single fixed setting across a mixed corpus.
How Netray Approaches Chunking for Client RAG Systems
Netray does not default to one chunking strategy across every client corpus, because a support knowledge base and a scanned engineering drawing archive have almost nothing in common structurally. Our DataRay ingestion pipeline classifies incoming documents by type and applies structural chunking to formally structured content, semantic chunking to long mixed-topic prose, and fixed-size chunking as the fallback, all validated against a client-specific golden evaluation set before anything ships. For manufacturers with decades of scanned specifications and drawings, we pair structural chunking with the layout-aware and vision-language techniques covered in our multimodal RAG guide.
Frequently Asked Questions
What is late chunking and why does it improve RAG accuracy?
Late chunking embeds an entire document with a long-context embedding model first, then pools the token-level embeddings into chunk-sized vectors afterward, rather than embedding each chunk in isolation. Because every chunk's vector reflects the full document context, cross-references and pronouns resolve correctly, which improves recall on documents with heavy internal references. It requires a long-context embedding model and costs more to re-embed when documents change.
Is semantic chunking always better than fixed-size chunking?
No. Semantic chunking helps most on long, mixed-topic documents where a fixed window would blend unrelated content, but it adds embedding compute at ingestion time and requires tuning a similarity threshold per corpus. For homogeneous, single-topic documents, fixed-size chunking with modest overlap performs comparably at a fraction of the implementation and compute cost, so validate the improvement against your golden set before adopting it.
What chunk size works best for technical and engineering documents?
Structural chunking, splitting along the document's own sections, tables, and headers, generally outperforms any fixed token count for technical and engineering documents. Where a fixed size is still needed as a fallback within a section, 300 to 500 tokens with metadata identifying the section, drawing sheet, or table name tends to balance retrieval precision against the surrounding context the generator needs.
How much chunk overlap should a production RAG system use?
A 10 to 20 percent overlap is a reasonable default that prevents information near a chunk boundary from being lost entirely from both neighboring chunks. Higher overlap increases storage and retrieval cost with diminishing accuracy returns beyond roughly 20 percent. Structural chunking that splits along natural document boundaries often needs little or no overlap, since the boundaries themselves are already meaningful breakpoints.
Key Takeaways
- 1Fixed-Size Chunking: The Baseline Everyone Starts With: Fixed-size chunking splits text into windows of a set token or character count, typically 256 to 512 tokens, with a 10 to 20 percent overlap so a sentence spanning a boundary is not lost entirely from either chunk. It is cheap, deterministic, and easy to reason about, and it works well on homogeneous prose like policy documents, articles, and most support documentation.
- 2Semantic Chunking: Splitting on Meaning Boundaries: Semantic chunking embeds sentences or small groups of sentences, measures the similarity between consecutive segments, and inserts a chunk boundary where similarity drops sharply, on the theory that a topic shift is a natural place to split. It genuinely helps on long-form, mixed-topic documents like meeting transcripts, multi-section reports, or long email threads where a fixed window would otherwise mash two unrelated topics into one chunk.
- 3Structural Chunking: Respecting Document Shape: Structural chunking splits along the document's own structure: markdown headers, table boundaries, code blocks, or, for engineering and ERP documentation, sections like a title block, bill of materials, or revision history. This is usually the best default for technical corpora because the document author already encoded meaningful boundaries, and ignoring them is what causes a 400-token fixed window to cut a specification table in half.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
RAG Chunking Strategy Calculator
Turn corpus size, chunk length, and overlap into a concrete chunk count, embedding cost, and vector storage footprint before you build the ingestion pipeline.
Free ToolPrivate RAG Corpus Sizing Calculator
Estimate chunk counts, vector index storage, raw text volume, and embedding compute time before you build a private retrieval system over your document estate.
Free ToolDocument Ingestion Pipeline Estimator
Estimate total pipeline time from document count, OCR share, and embedding throughput, so ingestion timelines stop being a guess in the project plan.
Terms used in this article
Not sure which chunking strategy fits your document set? Netray will run a chunking evaluation against your real corpus and show you the recall numbers before you commit to an architecture.
Related Resources
Enterprise RAG Architecture: The Full 2026 Blueprint
A practitioner's blueprint for enterprise RAG in 2026: ingestion, chunking, embedding, retrieval, rerank, generation, and the eval loop that keeps it honest.
AI & AutomationRAG Evaluation Metrics: Recall@k, MRR, Faithfulness, and More
The RAG evaluation metrics that matter: recall@k and MRR for retrieval, faithfulness and answer relevance for generation, and how to build the eval loop.
AI & AutomationMultimodal RAG Over Engineering Drawings and Scanned Documents
Multimodal RAG for manufacturers: retrieving from CAD drawings, scanned specs, and PDFs with embedded tables, using vision embeddings and layout-aware parsing.