INT8 is usually close to free; INT4 is a real trade you have to measure. Quantization stores weights at lower precision, cutting memory roughly in half per halving of bit width. The loss is never evenly spread — a small number of outlier values carry disproportionate importance, and naive schemes destroy them first.
The number everyone quotes is memory. A 70B parameter model at 16-bit needs around 140 GB just for weights; at 4-bit it needs roughly 35 GB. That is the difference between a multi-GPU deployment and a single card.
The number nobody quotes is which capability you gave up to get there, because averaged benchmarks hide it almost perfectly.
- ~2xMemory reduction per halving of bit width, before overhead
- 4-bitPrecision at which QLoRA fine-tunes a 65B model on one 48GB GPU
- NF4Data type its authors call optimal for normally distributed weights
- 65BModel size QLoRA reports fitting on a single 48GB card
- 16-bitBaseline performance QLoRA reports preserving at 4-bit
What does quantization actually do?
It maps a continuous range of floating point values onto a small set of discrete levels, then stores the index of the level instead of the value.
An 8-bit integer offers 256 levels. A 4-bit integer offers 16. The mapping needs a scale factor, and sometimes a zero point, to translate back during computation.
Everything interesting follows from one question: how do you choose the range being mapped?
Figure 1 — Why a single outlier ruins a whole block
Well-behaved block
Values cluster tightly, say between -0.4 and 0.4. The 16 available levels spread across a narrow range, so each level is fine-grained.
Rounding error stays small relative to the values themselves.
Low error
Block containing one outlier
A single value at 18.0 forces the range to span -18 to 18. The same 16 levels now sit far apart.
Every ordinary value in the block is crushed toward the nearest coarse level, and most of the block’s information is lost to serve one number.
Catastrophic error
This is the central problem in low-bit quantization. Transformer activations develop consistent outlier features at scale, and they matter — they cannot simply be clipped. Every serious method is, at heart, a different answer to what to do about them.
How do the main approaches differ?
| Approach | Core idea | Costs you |
|---|---|---|
| Per-tensor scaling | One scale factor for an entire tensor | Cheapest and most fragile; one outlier degrades everything |
| Per-channel or per-block | Separate scale per row, column or fixed-size block | Small metadata overhead, large accuracy gain |
| Mixed precision | Keep outlier dimensions in higher precision, quantize the rest | Implementation complexity; irregular memory access |
| Calibrated post-training | Use sample data to choose ranges that minimise error | Needs representative calibration data |
| Quantization-aware training | Simulate quantization during training so weights adapt | Requires a training run; best quality |
| Double quantization | Quantize the scale factors themselves | Extra decode step for further memory savings |
Block-wise scaling is the single highest-value idea here, because it confines the damage an outlier can do to its own block rather than letting it contaminate an entire tensor.
QLoRA combines several of these. Dettmers et al. (arXiv:2305.14314, May 2023) introduced 4-bit NormalFloat, described as information theoretically optimal for normally distributed weights, alongside double quantization to shrink the footprint of the quantization constants and paged optimizers to absorb memory spikes.
Their reported result is the strongest published evidence that 4-bit need not be lossy in practice: 4-bit NormalFloat preserving full 16-bit fine-tuning task performance, with 65B fine-tuning fitting on a single 48GB GPU.
What do you actually lose?
Not general fluency. That survives aggressive quantization and is why quantized models demo well.
What degrades first tends to be the capabilities with the least redundancy in the weights.
- Long multi-step reasoning, where small per-step errors compound across a chain
- Rare vocabulary and proper nouns, which depend on precise, sparsely-used weights
- Low-resource languages, for the same reason
- Exact arithmetic, where being nearly right is being wrong
- Long-context recall, particularly for details deep in the middle of the input
- Structured output conformance under complex schemas
Watch for this
Perplexity is the standard quantization metric and it is close to useless for this decision. It measures average next-token prediction over generic text, which is exactly the capability quantization preserves best. A model can hold near-identical perplexity while losing several points of accuracy on multi-step reasoning. Measure the task you actually run.
Which precision should you choose?
| Precision | Memory vs 16-bit | Use when |
|---|---|---|
| 16-bit | Baseline | Quality is the constraint and memory is not |
| 8-bit | About half | Default for production serving; degradation is usually negligible |
| 4-bit | About a quarter | Memory-bound deployment, or fine-tuning on one card |
| Below 4-bit | Less than a quarter | Research and extreme constraints; expect real quality loss |
The practical rule most teams converge on: go to 8-bit without much deliberation, and treat 4-bit as a decision requiring evidence from your own evaluation.
Below 4-bit, the honest framing is that you are trading capability for feasibility. Sometimes that trade is correct — a 4-bit larger model frequently beats an 8-bit smaller one at the same memory budget — but it should be a measured choice, not a default.
Does quantization make inference faster?
Often, but not for the reason people assume, and not always.
Text generation is usually memory-bandwidth bound rather than compute bound. Each token requires reading the weights, so halving weight size roughly halves the bytes moved, and throughput improves.
The exception is when dequantization overhead eats the gain. If weights must be unpacked to a higher precision before the matrix multiply, and the hardware has no native kernel for your format, you can end up slower than 16-bit despite using less memory.
Benchmark on your actual hardware. Format support varies enough that transferring a result between GPU generations is unsafe.
How should you validate a quantized model?
Figure 2 — Validation order, cheapest signal first
- STEP 1Task evalRun your own golden dataset at 16-bit and at the target precision
- STEP 2Stress the weak spotsMulti-step reasoning, arithmetic, rare entities, long context
- STEP 3Measure real latencyOn your hardware, at your batch size, not a published figure
- STEP 4Decide at equal memoryCompare a quantized larger model against a smaller one at the same footprint
Step 4 is the comparison that matters and the one most often skipped. The real question is never “is 4-bit worse than 16-bit” — it obviously is. It is “what is the best model I can fit in the memory I have.”
Where do teams go wrong?
Trusting a published benchmark table
Quantization degradation is task-dependent and model-dependent. A table showing negligible loss on one model family tells you very little about yours, and nothing about your workload.
Quantizing everything uniformly
Not all layers tolerate quantization equally. Attention projections and the final output layer are frequently more sensitive than feed-forward blocks. Uniform application is simple and leaves quality on the table.
Ignoring the KV cache
At long context and large batch sizes, the key-value cache can consume more memory than the weights. Quantizing weights to 4-bit while leaving a 16-bit cache solves the smaller half of the problem and puzzles teams who expected bigger savings.
What do experienced teams do differently?
They pick the model size first, then quantize to fit, rather than picking a model and asking how small it can get.
The reframing matters because it surfaces the actual trade. At a fixed 24 GB budget, the choice is between a large model at 4-bit and a mid-size model at 8-bit, and that is an empirical question with a different answer per task.
They also keep a 16-bit reference deployment available, even if it is slow and rarely used. When output quality is questioned months later, being able to run the same input through an unquantized model settles the argument in minutes.
A short glossary
- Scale factor
- The multiplier translating a stored integer level back to an approximate original value.
- Block-wise quantization
- Assigning a separate scale to each fixed-size group of weights, containing the damage from outliers.
- Outlier feature
- A dimension carrying values far larger than typical, which dominates range selection and resists naive quantization.
- NF4
- 4-bit NormalFloat, a data type designed for weights that follow a normal distribution.
- Double quantization
- Quantizing the quantization constants themselves to reclaim additional memory.
- Post-training quantization
- Quantizing an already-trained model, optionally using calibration data, without further training.
Key takeaways
- Quantization halves memory per halving of bit width, but the accuracy loss is unevenly distributed across capabilities.
- Outlier values dominate range selection, which is why block-wise scaling outperforms per-tensor scaling so decisively.
- QLoRA reports 4-bit NormalFloat preserving full 16-bit fine-tuning task performance, with 65B fitting on a single 48GB GPU.
- Multi-step reasoning, exact arithmetic, rare entities and long-context recall degrade before general fluency does.
- Perplexity is a poor proxy for quantization damage because it measures what quantization preserves best.
- Speed gains come from reduced memory bandwidth, and can be erased by dequantization overhead on unsupported hardware.
- The decision worth running is which model fits best in your memory budget, not whether 4-bit is worse than 16-bit.
Frequently asked questions
Is INT8 quantization safe for production?
For most workloads, yes. Degradation at 8-bit is typically small enough to be within noise on task evaluations, and memory drops by roughly half. Validate on your own golden dataset rather than assuming, particularly if your task involves exact arithmetic or long multi-step reasoning.
How much quality do I lose at 4-bit?
It depends heavily on method and task. QLoRA reports that 4-bit NormalFloat preserved full 16-bit fine-tuning task performance, but that result used careful block-wise handling. Naive 4-bit schemes lose considerably more, and reasoning-heavy tasks degrade before general fluency shows any change.
Why do outliers matter so much in quantization?
Because the quantization range must cover the largest value in the group. One value far from the rest stretches that range, so the limited levels spread thinly and every ordinary value in the block rounds to a coarse approximation. Block-wise scaling limits how far this damage spreads.
Does quantization always make inference faster?
No. Generation is usually memory-bandwidth bound, so smaller weights normally raise throughput. But if your hardware lacks a native kernel for the format, weights must be dequantized before computation, and that overhead can cancel or exceed the bandwidth saving. Benchmark on your own hardware.
Should I quantize weights or the KV cache first?
Start with weights, then measure. At long context lengths and large batch sizes the key-value cache can exceed weight memory, so teams quantizing only weights often see smaller savings than expected. Profile actual memory use at your real context length before choosing.
Is a quantized large model better than a small unquantized one?
Frequently yes at equal memory, but it is genuinely task-dependent. Larger models carry more redundancy and tend to absorb quantization damage better. Run both at your memory budget against your own evaluation set, since this comparison is the actual decision and is rarely published for your workload.
What is the difference between post-training quantization and quantization-aware training?
Post-training quantization converts an already-trained model, optionally using calibration data to pick ranges. Quantization-aware training simulates the precision loss during training so weights adapt to it. The second yields better quality at low bit widths and requires a training run you may not be able to afford.
Can I fine-tune a quantized model?
Yes, which is precisely what QLoRA does: gradients backpropagate through a frozen 4-bit base into low-rank adapters kept at higher precision. The base weights stay quantized and unchanged, so memory stays low while the trainable adapter avoids the precision loss.
References
- Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314, submitted 23 May 2023. Source for NF4, double quantization, paged optimizers, and the 65B on 48GB result.
- Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. arXiv:2208.07339. Documents emergent outlier features and mixed-precision decomposition.
- Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323.
