Why your training dataset won't load or trains incorrectly, and how to fix the format
KeyError: 'text' (or 'messages', 'prompt', 'completion')
Also appears as
- ValueError: Unable to create tensor, you should probably activate truncation and/or padding
- jsonlines.InvalidLineError: line contains invalid json
Short answer
Dataset format errors happen because the trainer expects a specific schema (either a messages list of role/content dicts, or a prompt/completion pair, or a single text field) and your JSONL doesn't match it, because samples are missing an EOS token so the model never learns to stop generating, or because a fixed max_length silently truncates long examples and cuts off labels partway through the intended response. Confirm your exact schema against what SFTTrainer or your data collator expects before training.
Affects: SFT with TRL's SFTTrainer, custom HuggingFace Trainer scripts, any JSONL-based instruction dataset
Fastest path to a correctly formatted dataset
- 1Print the first 3 raw rows of your dataset and identify which schema they actually use: text, {"prompt", "completion"}, or {"messages": [{"role", "content"}, ...]}.
- 2Match that schema to what your trainer expects; TRL's SFTTrainer wants either a text field or a messages field it can pass through apply_chat_template, not an arbitrary custom key.
- 3Confirm every example ends with the tokenizer's EOS token before truncation is applied, by checking input_ids[-1] equals tokenizer.eos_token_id after tokenization for several samples.
- 4Check your max_length or max_seq_length setting against your longest examples' token counts, since silent truncation cuts labels for the longest responses without any warning.
- 5Validate the entire file parses as JSONL by loading it independently with the json module line by line before handing it to the trainer, to catch malformed lines early.
How to confirm this is your problem
- Trainer raises a KeyError naming a field like 'text', 'messages', 'prompt', or 'completion' that it expected but didn't find
- Loading the dataset raises a JSON parsing error on a specific line number
- Model never produces an end-of-sequence token during generation, running until max_new_tokens is hit every time
- Training completes without error but the model performs poorly specifically on longer examples
- Tokenization warns about sequences exceeding max_length and being truncated
Root causes and fixes
Dataset schema does not match what the trainer expects
TRL's SFTTrainer, a custom HuggingFace Trainer, and raw JSONL loading each expect a specific field name and structure. A dataset built with a custom key like {"input", "output"} will fail to map to a trainer expecting {"prompt", "completion"} or a text field, because there is no field-name auto-detection happening by default in most training scripts.
Fix: Standardize your dataset to one supported schema (text field with the full formatted conversation, or messages as a list of role/content dicts) and rename keys during a preprocessing pass before training rather than trying to make the trainer flexible to arbitrary key names.
python -c "import datasets; ds=datasets.load_dataset('json', data_files='train.jsonl'); print(ds['train'][0])"Mixing messages-style and prompt/completion-style examples in one file
If some rows use a messages list of role/content turns and others use a flat prompt/completion pair (often from merging datasets from different sources), the data loading and formatting function only handles whichever schema it was written for, and rows in the other format either raise a KeyError or get silently formatted incorrectly, producing garbage training examples that don't error but still hurt the model.
Fix: Convert every example to a single consistent schema during a preprocessing step, and validate the converted dataset by rendering several examples through your formatting function and manually reading the output text.
Missing EOS token at the end of the response
If the response portion of a training example is not terminated with the tokenizer's EOS token before tokenization, the model never sees an example of when generation should stop, so it learns to keep generating past the intended answer indefinitely. This is a frequent issue with tokenizers that add EOS at the sequence end by default in some modes but not others depending on how you call the tokenizer.
Fix: Explicitly append the EOS token to every response string before tokenizing (or confirm your tokenizer call includes add_special_tokens correctly), and verify by checking that input_ids[-1] equals tokenizer.eos_token_id for training examples.
python -c "print(tokenizer.eos_token, tokenizer.eos_token_id)"
Fixed max_length silently truncates long examples, cutting off labels
When a training example's token count exceeds max_length, standard truncation cuts from the end, which often removes part or all of the intended response (the labels), leaving the model to train on a prompt with a partial or entirely missing target. Because truncation happens silently by default, this can go unnoticed for a significant fraction of a dataset's longer examples.
Fix: Measure the token length distribution of your dataset before training and set max_length to cover at least the 95th-99th percentile, or filter out (rather than silently truncate) examples that exceed a length you've decided is acceptable to drop.
python -c "import datasets; ds=datasets.load_from_disk('train'); lens=[len(x['input_ids']) for x in ds]; import statistics as s; print(max(lens), s.quantiles(lens, n=100)[94])"Malformed JSONL with invalid lines or inconsistent encoding
A JSONL file with a stray trailing comma, an unescaped quote inside a string field, or mixed encodings (a line saved with a different encoding than the rest of the file) can cause the JSON parser to fail on that specific line, and depending on the loader, this either halts the entire load or silently skips the malformed rows without informing you how many were dropped.
Fix: Validate the file by parsing it independently line by line with Python's json module before handing it to any training framework, logging the line number and error message for any line that fails so you can fix the source data directly.
python -c "import json\nbad=0\nfor i,l in enumerate(open('train.jsonl', encoding='utf-8')):\n try: json.loads(l)\n except Exception as e: print(i, e); bad+=1\nprint('bad lines:', bad)"Diagnostic commands
Print raw dataset rows to confirm schema
python -c "import datasets; ds=datasets.load_dataset('json', data_files='train.jsonl'); print(ds['train'][0]); print(ds['train'][1])"Confirms exactly which keys exist in your data (text, messages, prompt/completion, or something custom) so you can match your formatting function to what's actually there rather than what you assumed was there.
Check that examples end with EOS after tokenization
python -c "ids=tokenizer(example_text)['input_ids']; print(ids[-3:], tokenizer.eos_token_id)"
The last token id should match tokenizer.eos_token_id. If it doesn't, the model is not seeing where to stop and will likely run on during generation.
Measure token length distribution against max_length
python -c "lens=[len(tokenizer(x['text'])['input_ids']) for x in ds]; print(max(lens))"
If the maximum length substantially exceeds your configured max_length, a meaningful fraction of your dataset is being truncated, likely cutting into labels for your longest and often most information-dense examples.
Validate JSONL parses line by line
python -c "import json; [json.loads(l) for l in open('train.jsonl', encoding='utf-8')]"Raises immediately at the first invalid line with its line number if any row is malformed JSON, letting you fix the exact source row rather than debugging a confusing downstream trainer error.
Stopping it from happening again
- Pick one dataset schema (messages or prompt/completion or text) at project start and enforce it with a validation script run before every training job.
- Add an automated check for EOS token presence and length-distribution reporting as a required preprocessing step, not an optional one.
- Never silently truncate: log or reject examples that exceed max_length so you know exactly how much data is affected.
- Keep a small set of held-out format sanity-check examples (including edge cases like very long responses) that you re-verify after any preprocessing script change.
- Version your dataset preprocessing scripts alongside the raw data so format changes are traceable.
When this becomes an architecture problem
If your organization is merging training data from many sources (support tickets, documentation, chat logs) with inconsistent formats on an ongoing basis, ad hoc format-fixing scripts stop scaling and you need a proper data pipeline with schema validation built in. If dataset quality issues keep surfacing as training problems rather than being caught before training starts, that's a signal to invest in a dedicated data preparation and validation stage as part of your fine-tuning process.
Frequently asked questions
What dataset schema does TRL's SFTTrainer expect?
SFTTrainer can work with a text field containing the fully formatted training string, or a messages field structured as a list of role/content dictionaries that gets passed through the tokenizer's chat template. It does not automatically understand arbitrary custom field names like input/output without a formatting function you provide.
Why does my fine-tuned model never stop generating?
This almost always means training examples were missing the EOS token at the end of the response, so the model never learned an example of where a response should end. Check that input_ids for your training examples end with tokenizer.eos_token_id and add it explicitly if it's missing.
Does truncation happen silently during tokenization?
Yes, by default, tokenizers truncate to max_length without raising an error, simply cutting tokens from the end (or configured side) of the sequence. This can silently remove the response portion of long training examples unless you explicitly check length distributions or filter examples before training.
How do I check if my JSONL file has invalid lines?
Parse the file independently with Python's json module, line by line, wrapped in a try/except that logs the line number and error for any failure. This isolates malformed rows before they reach a training framework, where the resulting error message is often less specific about which line caused the problem.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
Fine-Tuning Dataset Size Estimator
Estimate the number of training examples, total tokens, and human curation hours needed for a fine-tuning dataset based on task complexity and quality bar.
Free ToolLoRA Fine-Tuning Cost Calculator
Turn model size, dataset tokens, epochs, and rank into a GPU-hour and dollar estimate for a LoRA fine-tuning run on rented or owned hardware.
Free ToolFine-Tuning GPU-Hours Estimator
Estimate GPU-hours for LoRA, QLoRA, and full fine-tuning on the same model size and dataset, so you can compare method tradeoffs before choosing.
Free ToolModel Context Window Planner
Allocate a fixed context window across system prompt, retrieved chunks, conversation history, and reserved output, then see exactly how much retrieval headroom is left.
Related problems
Tokenizer padding and truncation errors during training
Padding errors during training happen because many base models ship without a defined pad token at all, because the common workaround of setting pad_token equal to eos_token teaches the model that end-of-sequence and padding look identical (so it can learn to never emit a real stop signal), or because left-padding is used when right-padding was needed for the training collator, or vice versa. Add a distinct pad token when possible, and always right-pad for causal LM training.
Training loss not decreasing during fine-tuning
Training loss that stays flat is most often caused by a LoRA adapter that targets the wrong modules (missing q_proj/k_proj/v_proj/o_proj), a learning rate that is too low for LoRA or too high and bouncing, or labels that were never masked so the model is trying to learn the prompt tokens as if they were random noise. Check target_modules first, then the label mask, then the learning rate.
Loss becomes NaN during fine-tuning
NaN loss during training is most often caused by fp16 numeric overflow in gradients or activations, which bf16 avoids because of its wider exponent range. Other common causes are a learning rate spike (especially right after warmup), a small number of corrupt or malformed training samples, and unsafe division or log operations in a custom loss function. Switch to bf16 first if your hardware supports it, then check for corrupt samples and unstable LR.
Corrupted model checkpoint fails to load or loads with garbage weights
A corrupted checkpoint means the bytes on disk do not match the original artifact the model author published, whether from an interrupted download, a bad copy between systems, disk-level bit rot, or a failed write during a save operation. There is no reliable way to repair a corrupted deep learning checkpoint; the fix is always to verify the file against a known-good hash or size and re-obtain a clean copy, then build a verification step into your pipeline so the same failure does not silently recur.
GuideBuilding Fine-Tuning Datasets From Enterprise Data
Build fine-tuning datasets from enterprise data: instruction formats, deduplication methods, PII scrubbing, and quality filtering that actually works.
GuideFine-Tuning Failure Modes: What Actually Goes Wrong
Fine-tuning failure modes that actually derail enterprise projects: catastrophic forgetting, eval overfitting, data leakage, and how to catch each one.
GuideLoRA vs QLoRA: Choosing the Right Fine-Tuning Method
LoRA vs QLoRA for enterprise fine-tuning: rank and alpha choices, real VRAM math by model size, and when each method actually wins.
GuideHow to Evaluate a Fine-Tuned Model Before Production
Evaluate a fine-tuned model before production: held-out eval sets, task-specific metrics, calibrated LLM-as-judge setups, and regression testing.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.