Why safetensors throws HeaderTooLarge or InvalidHeaderDeserialization, and how to fix it
safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge
Also appears as
- safetensors_rust.SafetensorError: Error while deserializing header: InvalidHeaderDeserialization
- SafetensorError: Metadata not found
- Error while deserializing header: HeaderTooLarge (attempted to allocate ... bytes)
Short answer
A safetensors HeaderTooLarge or InvalidHeaderDeserialization error means the file's first bytes are not the expected binary length-prefixed JSON header, almost always because the file on disk is not the actual model weights but a truncated partial download or a small Git LFS pointer text file that was never smudged into the real binary. The fix is to verify the file size matches what the Hub reports and re-download it properly, either with huggingface_hub or with git lfs pull, not to try to repair the file in place.
Affects: Any model shipped in .safetensors format, transformers/safetensors of any recent version, most common after interrupted downloads or git clone without LFS smudge
Fix it in a few minutes
- 1Check the file size: ls -la for the .safetensors file. If it is a few hundred bytes to a few KB, it is an LFS pointer or a failed download, not real weights.
- 2If you used git clone on a HuggingFace repo, run git lfs install once, then git lfs pull inside the repo to fetch the actual binary content.
- 3If you used snapshot_download or from_pretrained, delete the specific corrupted file from the local cache (under ~/.cache/huggingface/hub/) and retry, letting it re-download cleanly.
- 4Prefer huggingface_hub's snapshot_download or hf download with resume_download enabled instead of raw git clone for large model repos.
- 5After re-download, confirm the file loads: python -c "from safetensors import safe_open; safe_open('model.safetensors','pt')".
How to confirm this is your problem
- Loading fails specifically on one shard file (model-00003-of-00007.safetensors) while others loaded fine
- The problematic file's size on disk is dramatically smaller than sibling shard files or than the size listed on the HuggingFace repo page
- cat or head on the file shows plaintext like 'version https://git-lfs.github.com/spec/v1' instead of binary content
- The error occurs consistently on every load attempt from that specific local file, not intermittently
Root causes and fixes
File on disk is an unsmudged Git LFS pointer, not the real binary weights
HuggingFace repos store large files via Git LFS. A plain git clone without git-lfs installed, or with LFS smudge filters disabled (common in CI containers, Docker builds, or air-gapped mirrors), downloads only the tiny LFS pointer text file for each large asset. safetensors then tries to parse that pointer's opening bytes as a binary header length and fails immediately.
Fix: Install git-lfs (apt-get install git-lfs or equivalent), run git lfs install, then git lfs pull from inside the cloned repo to replace every pointer file with the real binary content.
git lfs install git lfs pull git lfs ls-files
Download was interrupted or truncated (network drop, disk full, killed process)
safetensors reads a 8-byte length prefix followed by a JSON header of that exact length. If the download stopped partway through, the file ends abruptly and either the length prefix points past the actual file size or the JSON itself is cut off mid-object, producing HeaderTooLarge or a deserialization failure.
Fix: Delete the partial file and re-download with a resumable client. huggingface_hub's snapshot_download and hf download resume broken transfers automatically by default in current versions; verify final file size against the repo's listed size before trusting it.
rm ~/.cache/huggingface/hub/models--*/blobs/<partial-file>
python -c "from huggingface_hub import snapshot_download; snapshot_download('org/model')"Disk quota, out-of-space, or filesystem error silently truncated the write
On a full disk or a filesystem hitting a quota limit, some write paths fail silently or truncate rather than raising an obvious I/O error, leaving a partially-written safetensors file that looks superficially present but has an incomplete or absent header.
Fix: Check available disk space before large downloads (a 70B model in FP16 needs well over 140GB), and after download always verify file size against the expected value rather than assuming presence means completeness.
df -h du -sh ~/.cache/huggingface/hub/models--org--model
File was copied or synced with a tool that mangled binary content (e.g. text-mode transfer, wrong rsync flags, antivirus quarantine)
Certain file transfer tools or misconfigured sync jobs (FTP in ASCII mode, some corporate proxy content scanners, misconfigured rsync excludes) can alter or truncate binary files during transfer between an internet-connected staging host and an air-gapped target, producing the same header corruption symptom.
Fix: Transfer model artifacts as a single verified archive (tar plus a checksum file) rather than ad hoc file-by-file copies, and validate the checksum on the receiving side before use.
sha256sum model.safetensors
Diagnostic commands
Check actual vs expected file size
ls -la model.safetensors && curl -sI https://huggingface.co/org/model/resolve/main/model.safetensors | grep -i content-length
If the local size is smaller than the Content-Length reported by the Hub, the file is truncated or a pointer. They should match exactly for a good download.
Peek at the first bytes to detect an LFS pointer
head -c 200 model.safetensors
Readable text starting with 'version https://git-lfs.github.com' confirms this is an LFS pointer, not binary weights. Binary garbage in the terminal is expected and fine for a real safetensors file.
Validate the file opens correctly
python -c "from safetensors import safe_open; f=safe_open('model.safetensors','pt'); print(len(f.keys()))"A tensor key count greater than zero confirms the header parsed successfully. Any exception here reproduces the original bug and confirms the file itself, not the loading code, is the problem.
Stopping it from happening again
- Always use huggingface_hub's snapshot_download or hf download CLI for model weights instead of raw git clone, since it handles LFS content and resumable downloads correctly by default.
- For internal mirrors serving air-gapped environments, checksum every artifact (sha256) at mirror time and re-verify at the consuming end, never trust file presence alone.
- In CI/build images, explicitly install git-lfs and confirm git lfs pull before treating a cloned model repo as usable.
- Set a minimum free disk space check in deployment automation before starting multi-shard model downloads.
When this becomes an architecture problem
If this keeps recurring across an air-gapped mirror pipeline rather than being a one-off bad download, the fix is a proper internal artifact registry with checksum verification built into the transfer process, not repeated manual re-downloads. That is worth designing once as part of your model supply chain rather than firefighting per incident.
Frequently asked questions
Can I fix a truncated safetensors file without re-downloading?
No. Safetensors files are not designed to be patched or repaired; the header and tensor data must be complete and consistent. The only reliable fix is obtaining a complete copy of the file, either by re-downloading or by copying a verified good copy from another location.
Why did some shards load fine but one specific shard fails?
Multi-shard models download each file independently, so a network blip or interruption during one specific file's transfer will only corrupt that shard while others complete normally. Check that one file's size and checksum specifically against the Hub's listing.
Is this the same underlying issue as a 'corrupted checkpoint' error?
Yes, HeaderTooLarge is one specific manifestation of a corrupted or incomplete checkpoint file. See the general corrupted checkpoint troubleshooting page for broader verification steps covering both safetensors and other formats like GGUF or PyTorch .bin.
Does converting a .bin checkpoint to safetensors avoid this problem?
No, the format is not the cause here; an interrupted or LFS-pointer download of a .bin file fails just as badly, just with a different error message from torch.load. The root cause and fix (verify size, re-download or LFS pull) are the same regardless of weight format.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
Air-Gapped LLM Deployment Checklist
A practical control checklist for deploying and maintaining large language models in a fully air-gapped environment, from initial staging through ongoing patching and drift detection.
Free ToolOn-Prem AI Deployment Checklist
A 30-point pre-deployment checklist covering use cases, hardware, security, model operations, and rollout for self-hosted enterprise LLMs.
Related problems
Corrupted model checkpoint fails to load or loads with garbage weights
A corrupted checkpoint means the bytes on disk do not match the original artifact the model author published, whether from an interrupted download, a bad copy between systems, disk-level bit rot, or a failed write during a save operation. There is no reliable way to repair a corrupted deep learning checkpoint; the fix is always to verify the file against a known-good hash or size and re-obtain a clean copy, then build a verification step into your pipeline so the same failure does not silently recur.
HuggingFace model download is extremely slow or stalls partway through
Slow or stalled HuggingFace downloads are usually caused by huggingface_hub's default transfer path not using parallel chunked downloads, a corporate proxy or firewall throttling or dropping long-lived connections, or genuinely insufficient bandwidth for a hundreds-of-gigabytes model. Enable hf_transfer for a much faster Rust-based parallel downloader, rely on the client's built-in resume behavior rather than restarting from zero, and for regulated or air-gapped sites, download once and mirror internally instead of pulling repeatedly over the internet.
401/403 Unauthorized pulling a gated model from HuggingFace
A 401 or 403 on a gated HuggingFace repo means the request reached the Hub but was rejected for authorization, not because the model does not exist. The three real causes are: you have not accepted the model's license on the web page with the account tied to your token, the token exists but was created without read access to gated repos, or the token is valid but was never actually passed to the download call (no HF_TOKEN in the environment, no login run). Fix by accepting the license, generating a token with the right scope, and exporting it where the client library will find it.
Model loading fails offline or in an air-gapped environment despite having local files
Passing a local path to from_pretrained does not guarantee an offline load, because transformers and related libraries (tokenizers, some model configs, auto-mapping code) can still issue background network calls to check for updates, fetch a referenced remote component, or resolve auto_map entries that point back at the original HuggingFace repo. The fix is to set HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 explicitly, use a complete local snapshot directory (not just the weights file), and verify no config field still references a remote repo ID.
GuideSecuring Model Weights in the Enterprise
Secure model weights end to end: custody controls, encryption at rest, access policies, and exfiltration prevention for regulated AI deployments.
GuideAir-Gapped LLM Deployment Patterns That Actually Work
Air-gapped LLM deployment patterns that work: offline model transfer, update workflows, monitoring without telemetry, and CMMC-ready architectures.
GuideAir-Gapped Model Updates: A Patching Guide
Air-gapped model updates for enterprise AI: secure transfer procedures, hash verification, and staged rollout so patches never introduce risk.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.