Why fixed-size chunking destroys tables in your documents, and how to chunk them correctly
RAG answers are wrong for anything in a table even though the table is in the source document
Also appears as
- chunker splits a table row from its header
- retrieved chunk has numbers with no column labels
- table data returns garbled or headerless results in RAG
Short answer
Generic character-count or token-count chunking treats a document as an undifferentiated stream of text, so it routinely cuts a table's header row away from its data rows, leaving a retrieved chunk full of numbers with no column labels to explain what they mean. The fix is structure-aware chunking that detects table boundaries, keeps the header attached to every chunk of that table's rows, and never splits mid-row.
Affects: Any RAG pipeline ingesting documents with tables, specs sheets, price lists, or structured data, regardless of chunking library used.
Fastest path to correct table chunking
- 1Switch from naive character or token-count splitting to a structure-aware chunker that detects markdown tables, HTML tables, or PDF table regions as distinct units.
- 2For every chunk derived from a table, repeat the header row or a compact column-label summary at the top of the chunk, even if that means some duplication across chunks.
- 3Never split a table mid-row; if a table must be split across multiple chunks, split between row groups and repeat the header in each resulting chunk.
- 4Keep section headings attached to the content beneath them so a chunk of body text always retains its parent heading for context.
- 5Re-chunk and re-index any previously ingested documents that contain tables using the new structure-aware logic; old chunks cannot be patched in place.
How to confirm this is your problem
- Questions about specific table values get wrong or made-up answers even though the correct table is in the corpus
- Retrieved chunks contain rows of numbers or short codes with no visible column headers or labels
- A chunker's output for a table looks like a fragment of one row plus a fragment of the next, cut at an arbitrary character count
- Tables that fit entirely within one chunk answer correctly, but larger tables split across chunks fail
- Answers about a document section are correct for prose paragraphs but consistently wrong for anything under a heading followed by a table
Root causes and fixes
Chunking splits purely on character or token count with no awareness of document structure
A naive splitter has no concept of what a table is. It will cut through a table exactly where the count threshold falls, which is essentially random with respect to row and header boundaries, frequently separating column labels from the data rows that give them meaning.
Fix: Use a document parser that identifies table regions (markdown table syntax, HTML table tags, or a PDF layout model's table detection) as atomic or semi-atomic units, and chunk around them rather than through them.
Table headers are not repeated when a table is split across multiple chunks
Even structure-aware chunkers can produce multiple chunks from one large table if it exceeds the target chunk size. If the header row is only included in the first chunk, every subsequent chunk contains data rows with no column labels, making them uninterpretable in isolation to both a retriever and the generating LLM.
Fix: Explicitly repeat the header row, or a short synthesized column-label line, at the top of every chunk derived from the same table, accepting the modest token duplication cost as necessary overhead.
Section headings are stripped or separated from the body content during chunking
Splitters that operate on raw extracted text without heading awareness can place a heading at the end of one chunk and the actual content beneath it in the next chunk, so neither chunk alone conveys what the content is about, which hurts both embedding relevance and answer quality.
Fix: Use a heading-aware or markdown-aware splitter that keeps each heading attached to the paragraphs and tables beneath it within the same chunk, or at minimum repeats the heading text at the top of every chunk under that section.
PDF-to-text extraction flattens tables into unstructured text before chunking ever sees table boundaries
If the ingestion pipeline extracts raw text from a PDF without preserving table structure, the chunker downstream has no signal left to detect a table at all; it just sees text with irregular spacing, and structure-aware chunking has nothing to key off.
Fix: Use a PDF extraction tool or layout model that outputs structured table data (as markdown tables or a row/column data structure) rather than plain flattened text, so table structure survives into the chunking stage.
Diagnostic commands
Manually inspect chunks generated from a document section containing a table
python -c "for c in chunk_document('doc_with_table.pdf'): print(c)"If any printed chunk contains numeric table rows with no visible header text, this confirms the chunker is splitting tables without preserving headers.
Check whether the PDF extraction step preserves table structure at all
python -c "print(extract_text('doc_with_table.pdf')[table_start:table_end])"If the extracted text shows misaligned or run-together numbers with no clear row/column delimiters, the problem starts at extraction, before chunking even runs; fix extraction first.
Test retrieval and generation quality on a set of table-specific questions
python eval_table_qa.py --questions table_questions.jsonl
A large accuracy gap between prose questions and table questions confirms the issue is table handling specifically, not general retrieval quality, and justifies prioritizing structure-aware chunking over other fixes.
Stopping it from happening again
- Choose a document parser and chunker combination that explicitly supports table detection before building the rest of the pipeline around it.
- Always repeat table headers in every chunk derived from that table, and treat this as a hard rule in the chunking code, not an edge case.
- Test chunking quality specifically on documents containing tables as part of routine ingestion pipeline validation.
- Keep headings attached to their content through the entire pipeline, from extraction through chunking through indexing.
- Re-validate chunking output any time the source document format changes, such as a new PDF template or export tool.
When this becomes an architecture problem
If your corpus includes complex or nested tables, engineering drawings with embedded data tables, or scanned documents where table structure has to be reconstructed with a vision model rather than parsed directly, structure-aware chunking becomes a genuinely hard document-understanding problem rather than a chunker configuration change, and is worth scoping as its own workstream before it blocks the rest of the RAG rollout.
Frequently asked questions
Does duplicating table headers in every chunk waste too many tokens?
The overhead is usually small, typically one short row of column labels per chunk, and it is far cheaper than the cost of wrong answers on table-derived questions. For very wide tables, synthesize a compact column-label summary rather than repeating the full header verbatim if token budget is tight.
Should tables be chunked separately from surrounding prose?
Generally yes. Treating a table as its own chunk rather than merging it with surrounding paragraph text usually improves embedding quality, since the vector isn't diluted by unrelated prose, and improves answer quality, since the model gets a clean, self-contained table to reason over.
Can I fix this without re-ingesting my whole document set?
No. Chunking happens at ingestion time, so any documents already chunked with the old, structure-unaware method have corrupted table chunks baked into the vector index. You need to re-chunk and re-embed those documents, though you can usually detect which documents contain tables programmatically to avoid re-processing everything.
What is the best chunk size for a table-heavy document?
Size chunks around whole tables or logical row groups rather than a fixed token count; a small table might fit in one chunk with room for surrounding context, while a large table should be split at row-group boundaries with the header repeated, rather than forcing every chunk to hit the same token target.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
RAG Chunking Strategy Calculator
Turn corpus size, chunk length, and overlap into a concrete chunk count, embedding cost, and vector storage footprint before you build the ingestion pipeline.
Free ToolDocument Ingestion Pipeline Estimator
Estimate total pipeline time from document count, OCR share, and embedding throughput, so ingestion timelines stop being a guess in the project plan.
Free ToolRAG Accuracy Readiness Assessment
Score your retrieval-augmented generation system across eight dimensions that actually predict production accuracy, from chunking strategy to groundedness verification.
Related problems
PDF text extraction produces garbled or out-of-order text
Garbled PDF extraction almost always comes from using a text-layer extractor on a document that does not have the kind of text layer it expects: scanned or image-based PDFs need OCR, multi-column layouts need layout-aware extraction to preserve reading order, and ligature characters need Unicode normalization. Match the extraction method to the actual document type, and route scanned engineering drawings to a vision model instead of text extraction entirely.
RAG retrieves irrelevant or wrong documents
RAG retrieves the wrong documents most often because the embedding model used to index the corpus differs from the one used at query time, or because chunks are large enough that a single embedding averages away the passage that actually answers the question. Fix embedding consistency and chunk granularity first, then add a reranker and metadata filters before touching the LLM prompt.
RAG hallucinates even though the correct context was retrieved
RAG hallucination with good context in hand usually means the prompt never explicitly instructs the model to answer only from the provided passages, or the correct passage is buried in the middle of a long context window where attention is weakest. Add an explicit grounding instruction, place the most relevant passage first, resolve conflicting retrieved chunks before generation, and give the model an explicit abstain option.
GuideRAG Chunking Strategies: Fixed, Semantic, Structural, and Late
Compare RAG chunking strategies, fixed-size, semantic, structural, and late chunking, with concrete guidance on chunk size, overlap, and when each wins.
GuideEnterprise RAG Architecture: The Full 2026 Blueprint
A practitioner's blueprint for enterprise RAG in 2026: ingestion, chunking, embedding, retrieval, rerank, generation, and the eval loop that keeps it honest.
GuideRAG Evaluation Metrics: Recall@k, MRR, Faithfulness, and More
The RAG evaluation metrics that matter: recall@k and MRR for retrieval, faithfulness and answer relevance for generation, and how to build the eval loop.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.