LoRA freezes the pre-trained weights and trains small rank decomposition matrices injected into each Transformer layer. QLoRA adds 4-bit quantization of the frozen base, cutting memory far enough to fine-tune a 65B model on one 48GB GPU. Both produce adapters measured in megabytes, not gigabytes.
- 10,000xFewer trainable parameters than Adam full fine-tuning on GPT-3 175B
- 3xLower GPU memory requirement reported in the original LoRA paper
- 65BParameter model QLoRA fine-tunes on a single 48GB GPU
- 99.3%Of ChatGPT’s level reached by Guanaco on the Vicuna benchmark
- 24hSingle-GPU fine-tuning time behind that Guanaco result
The common assumption is that LoRA is the cheap approximation and full fine-tuning is the real thing. The original paper says otherwise. Hu et al. report LoRA performing “on-par or better than fine-tuning in model quality” on RoBERTa, DeBERTa, GPT-2 and GPT-3, while training up to 10,000 times fewer parameters.
If your LoRA run underperforms, the method is rarely the reason. What follows is the twelve-step sequence, the arithmetic behind rank and alpha, and a diagnostic tree for when it goes wrong.
What do you need before you start?
| Prerequisite | Why it matters | Time |
|---|---|---|
| A task with a clear eval | Without a metric you cannot distinguish adaptation from memorisation | 2-4 hours |
| 200-2,000 curated examples | QLoRA’s headline results came from small high-quality sets, not volume | 1-3 days |
| Held-out test split | Adapters overfit quickly and quietly | 30 minutes |
| Sufficient VRAM | LoRA cuts memory around 3x versus Adam full fine-tuning; QLoRA cuts far more | – |
| A prompted baseline number | Prompting often wins outright; you need to know what you are beating | 2 hours |
The baseline is the step teams skip, and it is the one that most often makes the whole project unnecessary.
How does LoRA actually work?
It exploits a property the authors call rank-deficiency in language model adaptation.
Instead of updating a weight matrix directly, LoRA holds it frozen and learns two small matrices whose product approximates the update. Rank r controls their size. Because the base never moves, one model can serve many adapters.
Figure 1 — Where the trainable parameters live
Why this is cheap: gradients are only computed for A and B. The optimizer state, which dominates memory in full fine-tuning, shrinks in proportion.
What does QLoRA add on top?
Three memory innovations, all documented in Dettmers et al. (arXiv:2305.14314, May 2023).
- 4-bit NormalFloat (NF4) — a data type the authors describe as information theoretically optimal for normally distributed weights.
- Double quantization — quantizing the quantization constants themselves to shave the average memory footprint further.
- Paged optimizers — absorbing the memory spikes that would otherwise crash a long run.
Gradients backpropagate through the frozen 4-bit base into the LoRA adapters. The paper reports this preserves full 16-bit fine-tuning task performance.
The resulting Guanaco models reached 99.3% of ChatGPT’s performance level on the Vicuna benchmark after 24 hours of fine-tuning on a single GPU. That result came from a study in which the authors fine-tuned more than 1,000 models across 8 instruction datasets.
The twelve steps
Figure 2 — The three phases, and where projects die
The ordering is deliberate: every step in phase one is cheap and can cancel the project. Nothing in phase two or three can rescue a task that never needed fine-tuning.
Steps 1-4: decide whether to fine-tune at all
- Write the eval first. Twenty to fifty examples with expected outputs. It goes wrong when the eval is written after training, because you will unconsciously fit it to whatever you got.
- Measure the prompted baseline. Run the base model with a strong prompt against that eval. Fine-tuning must beat this materially to justify its cost.
- Confirm it is a behaviour problem, not a knowledge problem. Fine-tuning teaches form, tone and task shape. It is a poor way to inject facts; retrieval handles that far better.
- Pick the smallest viable base model. Adapter quality tracks base quality, but a 7B model you can iterate on beats a 70B run you cannot.
Steps 5-8: configure the run
- Choose rank r. Start at 8 or 16. Higher rank adds capacity and memory. It goes wrong when teams raise rank to fix a run whose real problem is data.
- Set alpha. A common convention is alpha equal to rank, or twice the rank. Alpha scales the update, so changing it changes the effective learning rate.
- Select target modules. The original work targets attention projections. Extending to MLP layers raises capacity and cost. Add modules one group at a time.
- Enable NF4 with double quantization if memory is the binding constraint. Skip quantization when it is not, since it buys memory at some quality risk.
Steps 9-12: train, verify, ship
- Train for very few epochs. Small high-quality sets overfit fast. Two to three epochs is often already past optimal.
- Watch held-out loss, not training loss. Training loss falling while eval loss rises is the signature of memorisation.
- Verify against the step 1 eval and the step 2 baseline. This is how you know it worked, and it is the only comparison that matters.
- Merge or serve separately. Merging removes inference overhead. Keeping adapters separate lets one base serve many tasks.
How should you actually choose rank and alpha?
Rank is a capacity dial, not a quality dial. That distinction is where most tuning time gets wasted.
| Rank | Fits | Risk |
|---|---|---|
| 4-8 | Tone, format, output style, single narrow task | Underfits genuinely complex behaviour |
| 16-32 | Domain adaptation, structured output conformance | The usual sweet spot; little downside |
| 64+ | Multi-task adapters, large diverse datasets | Overfitting and catastrophic forgetting rise sharply |
Alpha is easier than the discourse suggests. Because the update is scaled by alpha divided by rank, changing rank while holding alpha fixed silently changes your effective learning rate.
Keep the ratio constant when you sweep rank. Otherwise you are running two experiments at once and cannot attribute the result to either.
How do you know it actually worked?
Three checks, run in this order.
- Regression check. Run general-capability prompts unrelated to your task. A tuned adapter that has degraded general reasoning is a bad trade you would otherwise discover in production.
- Held-out win rate. Compare adapter output against baseline output on unseen examples.
- Memorisation probe. Feed a training example with its ending removed. Verbatim completion means you memorised rather than generalised.
Watch for this
The QLoRA authors found that current chatbot benchmarks are not trustworthy for accurately evaluating chatbot performance. Treat any single benchmark number, including your own, as one weak signal rather than a verdict. Your own held-out eval on real task data beats a public leaderboard every time.
What goes wrong, and how do you diagnose it?
| Symptom | Likely cause | Fix |
|---|---|---|
| Loss barely moves | Learning rate too low, or alpha scaling the update to near zero | Raise LR an order of magnitude; check the alpha-to-rank ratio |
| Output repeats itself | Overtrained on a small set | Cut epochs; widen data variety |
| Strong on train, weak on held-out | Memorisation | Fewer epochs, lower rank, more diverse examples |
| Out-of-memory mid-run | Optimizer state spikes | Enable paged optimizers; cut sequence length before batch size |
| General reasoning degraded | Adapter overwriting broad behaviour | Lower rank; narrow target modules; reduce LR |
| Quantized run worse than 16-bit | Quantization compounding an already-marginal setup | Reproduce in 16-bit first, then quantize |
Figure 3 — Diagnostic tree for a disappointing run
Overfitting.Cut epochs first. Then lower rank. Then add data variety. In that order.
Nothing is learning.Check the alpha-to-rank ratio, then the learning rate, then confirm target modules are actually attached.
Catastrophic forgetting.Narrow target modules to attention only, lower LR, reduce rank.
Run the memorisation probe.Good curves still hide verbatim recall on small datasets.
Read the curves before touching config. Almost every wasted tuning day starts with someone changing rank before looking at which of these four states they are actually in.
When should you not fine-tune at all?
Four situations, and they cover more real projects than most teams expect.
- The requirement is factual accuracy. Retrieval grounds answers in sources you control and updates instantly. An adapter bakes facts in at training time and goes stale.
- The requirement changes weekly. Every change means a retrain and a re-verification cycle. A prompt change is a deploy.
- You have fewer than roughly 100 examples. Below that, few-shot prompting usually matches or beats an adapter, without the overfitting risk.
- Nobody has written the eval. Without it you cannot tell success from regression, and you will ship on vibes.
What do experienced practitioners do differently?
They treat data curation as the actual work and the training config as a formality.
The QLoRA authors fine-tuned more than 1,000 models across 8 instruction datasets and multiple model families, and their conclusion points the same way: small, high-quality datasets produced state-of-the-art results, even with smaller models than the previous best.
Two more habits worth stealing. They keep one adapter per task rather than training a single adapter to do several things, because narrow adapters are easier to evaluate and cheaper to retrain. And they version adapters against a pinned base model, since an adapter trained on one base is not portable to another.
A short glossary
- Rank (r)
- The inner dimension of the two trainable matrices, controlling how much capacity the adapter has.
- Alpha
- A scaling factor applied to the low-rank update. Because scaling is alpha over rank, the two must be tuned together.
- NF4
- 4-bit NormalFloat, QLoRA’s data type, described by its authors as information theoretically optimal for normally distributed weights.
- Double quantization
- Quantizing the quantization constants themselves to reduce average memory footprint further.
- Paged optimizer
- An optimizer that pages state between GPU and CPU memory to survive transient allocation spikes.
- Catastrophic forgetting
- Loss of previously held general capability as a model is adapted too aggressively to a narrow task.
Where this is heading
This is judgment rather than established fact: the gap between adapter tuning and full fine-tuning will keep narrowing until full fine-tuning becomes a niche operation performed mainly by base model builders.
The economics already point there. An adapter is megabytes, merges without latency cost, and can be swapped per request. Once serving infrastructure treats adapters as routine rather than exotic, keeping many small specialised adapters over one strong base becomes the obvious default.
The counter-argument deserves a hearing. Full fine-tuning still wins where a model must genuinely change its distribution rather than its behaviour, and low-rank updates are by construction a constrained approximation. If your task needs the model to become something different rather than to act differently, that constraint binds. I think that describes a shrinking minority of production work, but it is not nothing.
Key takeaways
- LoRA freezes base weights and trains injected rank decomposition matrices, cutting trainable parameters by up to 10,000 times versus Adam full fine-tuning on GPT-3 175B.
- The original paper reports LoRA matching or beating full fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3.
- LoRA adds no inference latency, because adapters merge back into the base weights.
- QLoRA’s NF4 quantization, double quantization and paged optimizers together fit 65B fine-tuning onto a single 48GB GPU.
- Guanaco reached 99.3% of ChatGPT’s Vicuna benchmark level after 24 hours on one GPU.
- Hold the alpha-to-rank ratio constant when sweeping rank, or you are unknowingly changing the learning rate too.
- Always measure a prompted baseline first; it frequently makes fine-tuning unnecessary.
Frequently asked questions
What rank should I use for LoRA?
Start at 8 or 16 and only raise it if held-out performance is capacity-limited rather than data-limited. Higher rank adds trainable parameters and memory. Most disappointing runs are fixed by better or more varied training examples, not by a larger rank value.
Is QLoRA worse than LoRA in quality?
The QLoRA paper reports that 4-bit NormalFloat quantization preserves full 16-bit fine-tuning task performance, so the intended answer is no. In practice quantization narrows margins, so reproduce a marginal setup in 16-bit before blaming quantization for a weak result.
How much data do I need to fine-tune with LoRA?
Less than most teams assume. The QLoRA authors found that fine-tuning on a small, high-quality dataset produced state-of-the-art results. A few hundred carefully curated examples usually beats tens of thousands of scraped ones for behaviour and format adaptation.
Can LoRA teach a model new facts?
Badly. Fine-tuning is effective at shaping tone, format and task behaviour, and inefficient at storing knowledge. For factual grounding, retrieval is the better tool, and the two combine well: retrieval supplies the facts while an adapter shapes how they are presented.
Do I need to merge the adapter before serving?
Only if you want zero inference overhead from a single model. Merging folds the low-rank update into the base weights permanently. Keeping the adapter separate costs a little at inference but lets one base model serve many task-specific adapters swapped at request time.
Why did my fine-tuned model get worse at everything else?
Catastrophic forgetting from too much adaptation pressure. Lower the rank, narrow the target modules to attention projections only, reduce the learning rate, and cut epochs. Then re-run a general-capability regression check against the base model to confirm the trade is acceptable.
What does alpha do in LoRA?
Alpha scales the low-rank update before it is added to the frozen weights, with the effective scale being alpha divided by rank. Raising alpha strengthens the adapter’s influence, which behaves much like raising the learning rate. Keep the ratio fixed when you change rank.
Can I use one LoRA adapter with a different base model?
No. The learned matrices are tied to the exact weights and layer shapes they were trained against. Moving an adapter to a different base, or even a different quantization of the same base, produces unpredictable output. Pin the base model version alongside every adapter you ship.
References
- Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685, submitted 17 June 2021, revised 16 October 2021.
- Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314, submitted 23 May 2023.
- Microsoft. LoRA reference implementation.
