LoRA, in five pictures
The 2021 paper that made it cheap to customise a huge model by training a tiny add-on, shown as five short animations.
In 2021, eight researchers at Microsoft found a way to customise a giant model without changing the model itself. You train a tiny add-on and leave everything else frozen. They called it LoRA.
1. Before: every custom version was a full copy
2. The idea: freeze the model, train two thin strips
1,536× fewer numbers to train
Trained: 98,304 numbers, in two thin strips.
3. How thin can the strips be?
4. The result: same quality, a sliver of the training
5. No slowdown when it runs
Why it matters today
- 01One shared model
- 02Many small add-ons
- 03Swap them per task
This is the usual way people customise open models today, because it fits on ordinary hardware. See fine-tuning.
Apple Intelligence, announced in 2024, runs one small model on the phone and swaps in small LoRA adapters for each job, such as summarising. One shared model with many small add-ons is this paper’s idea.
The details, if you want them
Who wrote it, and where to read it
LoRA: Low-Rank Adaptation of Large Language Models by Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen at Microsoft. First posted on 17 June 2021.
What was wrong with the alternatives
- Full fine-tuning makes a new model as large as the original for every task. For GPT-3 that is 175 billion parameters each time.
- Adapters, small extra layers placed inside the model, slow down every answer. The paper measured up to about 30% extra delay for single requests.
- Prefix tuning, which reserves part of the prompt for learned tokens, was hard to train and uses up space the real prompt needs.
The method, exactly
h = W₀x + BAx- W₀ is the original grid of numbers, kept frozen.
- B and A are the two thin strips. If W₀ is d by k, then B is d by r and A is r by k, where r (the rank) is small.
- A starts as random numbers and B starts as zeros, so at the start the add-on changes nothing.
- The paper applies this only to two of the grids inside each attention step and leaves the rest alone.
- The number 12,288 in picture 2 is the width the paper gives for GPT-3.
Why it should work
The authors build on earlier findings that big models, once trained, can be adjusted along surprisingly few directions. Their guess, in their words: “the change in weights during model adaptation also has a low intrinsic rank”.
All the results
- GPT-3: LoRA matched or beat full fine-tuning on all three tests tried: database queries (73.4 against 73.8, or 74.0 with a larger add-on), sentence logic (91.7 against 89.5) and summarising conversations.
- Memory while training GPT-3: from 1.2 TB down to 350 GB.
- RoBERTa and DeBERTa, two BERT-style models: LoRA trained 0.3 to 4.7 million numbers and scored level with or slightly above full fine-tuning, which trained 125 to 1,500 million.
- GPT-2: LoRA scored 70.4 on a text-writing test against 68.2 for full fine-tuning, training about a thousandth as many numbers.
- Storage: 100 customised GPT-3 models would take about 354 GB in total, against 35,000 GB as full copies.
The limits the authors list
- Once the strips are folded into the model, you cannot easily serve requests for different tasks in one batch.
- They do not expect a very thin add-on to work for every task, for example one in a different language from the original training.
Words the paper uses
- Low-rank: a big grid of numbers that can be rebuilt from two thin ones.
- Rank (r): how thin. Rank 1 is the thinnest possible.
- Adaptation: customising a trained model for a task.
- Checkpoint: a saved copy of a model.
- Inference latency: the delay when the model answers.
Next paper: Chain-of-Thought