What Is Quantization?
Also known as: Model Quantization, Weight Quantization
Definition
Quantization is the technique of storing a model's weights at lower numeric precision, for example 8-bit or 4-bit integers instead of 16-bit floats, which sharply reduces GPU memory use and often increases speed at a modest accuracy cost.
Quantization Explained
Models are usually trained in 16-bit floating point, so a parameter occupies two bytes and a 70-billion-parameter model needs about 140 GB just for weights. Quantization maps groups of weights onto a smaller integer range using a scale factor and sometimes an offset, then reconstructs approximate values during computation. At 8-bit the model halves to roughly 70 GB; at 4-bit it drops to roughly 35 to 40 GB including quantization metadata, which is the difference between four data center GPUs and one.
Not all quantization is equal. Naive round-to-nearest schemes degrade quality noticeably. Modern methods such as GPTQ and AWQ use a small calibration dataset to identify which weights are most sensitive and allocate precision accordingly, preserving accuracy far better at the same bit width. GGUF formats used by llama.cpp offer a spectrum of mixed-precision variants that keep critical layers at higher precision while compressing the rest aggressively.
The practical accuracy picture is reassuring at 8-bit and mostly reassuring at 4-bit. Eight-bit quantization is generally regarded as near-lossless for inference. Four-bit typically costs a small amount of quality, more visible on multi-step reasoning, precise arithmetic, and long-context tasks than on extraction, classification, or grounded question answering. Below 4 bits, degradation becomes pronounced quickly and is rarely worth the memory saved in an enterprise setting.
Speed effects are less intuitive than memory effects. Token generation on a single stream is usually memory-bandwidth bound, so moving fewer bytes per token makes quantized models faster. Under heavy batched load the workload shifts toward compute, and quantized models may lose some of that advantage depending on whether the hardware has native support for the arithmetic. Always benchmark at your real concurrency rather than trusting single-user numbers.
Why It Matters
- Turns a four-GPU deployment into a one-GPU deployment, which changes both capital cost and data center power and cooling requirements.
- Frees VRAM that can instead hold KV cache, directly increasing how many concurrent users a single server can support.
- Improves single-stream token throughput in memory-bandwidth-bound serving, making interactive plant-floor use feel responsive.
- Costs accuracy unevenly across tasks, so the decision must be validated against your own evaluation set rather than a public benchmark.
In Practice
A team compared a 4-bit 70B model against a 16-bit 13B model on the same single GPU. The quantized large model won clearly on grounded document question answering. On tasks needing exact multi-step arithmetic, the smaller full-precision model was the more reliable choice. Neither answer generalizes; the evaluation set decided it.
Frequently Asked Questions
How much accuracy do I lose with 4-bit quantization?
With a modern calibration-based method such as GPTQ or AWQ, the loss on typical enterprise tasks like extraction, classification, and grounded question answering is usually small enough to be hard to detect without careful measurement. Degradation is more noticeable on multi-step reasoning, precise arithmetic, and very long contexts, so validate on your own tasks before committing.
Should I run a larger quantized model or a smaller full-precision one?
For knowledge-heavy and language-heavy work, a larger quantized model generally beats a smaller full-precision model at the same memory budget, because parameter count contributes more than precision. For arithmetic-heavy or tightly structured output tasks the smaller high-precision model can be more reliable. Benchmark both on your workload rather than assuming.
Related Terms
GPU VRAM
GPU VRAM is the high-bandwidth memory physically attached to a graphics processor. For AI serving it must simultaneously hold the model weights, the key-value cache for every active request, and activation buffers, which makes it the binding constraint on most deployments.
Model Weights
Model weights are the numeric parameters inside a neural network, learned during training, that determine how input is transformed into output. They are the model: distributed as files, they encode everything the system has learned.
Inference
Inference is the phase in which a trained model is run to produce output from new input, as opposed to training, where the model's weights are being learned. In production, inference is where essentially all AI compute cost is spent.
Go Deeper
GPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Enterprise GPU Cluster Planning for AI Workloads
Plan an enterprise GPU cluster for AI workloads: H100 vs L40S sizing, networking, power, cooling, and cost models for on-prem LLM inference and training.
On-Prem LLM Deployment Architecture: Reference Guide
Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.
Working with Quantization in a live environment? Our engineers do this every day - and our AI agents automate most of it.