← Papers
5 min read

LoRA, in five pictures

The 2021 paper that made it cheap to customise a huge model by training a tiny add-on, shown as five short animations.

PapersFine-tuning
Watch it in about a minute

In 2021, eight researchers at Microsoft found a way to customise a giant model without changing the model itself. You train a tiny add-on and leave everything else frozen. They called it LoRA.

1. Before: every custom version was a full copy

File size of one customised GPT-3
  • The old way: a full copy 350 GB
  • LoRA: just the add-on 35 MB
The LoRA bar is there. It is 10,000 times smaller.

2. The idea: freeze the model, train two thin strips

Drag to change the size of the add-on

1,536× fewer numbers to train

Trained: 98,304 numbers, in two thin strips.

A model is made of big grids of numbers. LoRA leaves each grid alone and learns a small correction, stored as two thin strips. Multiplied together, the strips make a full-size correction.

3. How thin can the strips be?

Test score by thickness of the strips
  • Rank 1 73.4
  • Rank 2 73.3
  • Rank 4 73.7
  • Rank 8 73.8
  • Rank 64 73.5
Turning questions into database queries, on GPT-3. The thinnest add-on did about as well as one 64 times thicker.

4. The result: same quality, a sliver of the training

Same test, by how many numbers were trained
  • Whole model: 175 billion 73.8
  • LoRA: 4.7 million 73.4
  • LoRA: 37.7 million 74
Training also needed a third of the graphics memory and ran 25% faster.

5. No slowdown when it runs

Extra delay per answer, compared with the plain model
  • Earlier add-on methods up to +30%
  • LoRA 0%
Once trained, the strips are folded into the model's own numbers, so nothing extra runs.

Why it matters today

What came next
  1. 01One shared model
  2. 02Many small add-ons
  3. 03Swap them per task

This is the usual way people customise open models today, because it fits on ordinary hardware. See fine-tuning.

Where you see it today

Apple Intelligence, announced in 2024, runs one small model on the phone and swaps in small LoRA adapters for each job, such as summarising. One shared model with many small add-ons is this paper’s idea.

The details, if you want them

Who wrote it, and where to read it

LoRA: Low-Rank Adaptation of Large Language Models by Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen at Microsoft. First posted on 17 June 2021.

What was wrong with the alternatives
  • Full fine-tuning makes a new model as large as the original for every task. For GPT-3 that is 175 billion parameters each time.
  • Adapters, small extra layers placed inside the model, slow down every answer. The paper measured up to about 30% extra delay for single requests.
  • Prefix tuning, which reserves part of the prompt for learned tokens, was hard to train and uses up space the real prompt needs.
The method, exactly
h = W₀x + BAx
  • W₀ is the original grid of numbers, kept frozen.
  • B and A are the two thin strips. If W₀ is d by k, then B is d by r and A is r by k, where r (the rank) is small.
  • A starts as random numbers and B starts as zeros, so at the start the add-on changes nothing.
  • The paper applies this only to two of the grids inside each attention step and leaves the rest alone.
  • The number 12,288 in picture 2 is the width the paper gives for GPT-3.
Why it should work

The authors build on earlier findings that big models, once trained, can be adjusted along surprisingly few directions. Their guess, in their words: “the change in weights during model adaptation also has a low intrinsic rank”.

All the results
  • GPT-3: LoRA matched or beat full fine-tuning on all three tests tried: database queries (73.4 against 73.8, or 74.0 with a larger add-on), sentence logic (91.7 against 89.5) and summarising conversations.
  • Memory while training GPT-3: from 1.2 TB down to 350 GB.
  • RoBERTa and DeBERTa, two BERT-style models: LoRA trained 0.3 to 4.7 million numbers and scored level with or slightly above full fine-tuning, which trained 125 to 1,500 million.
  • GPT-2: LoRA scored 70.4 on a text-writing test against 68.2 for full fine-tuning, training about a thousandth as many numbers.
  • Storage: 100 customised GPT-3 models would take about 354 GB in total, against 35,000 GB as full copies.
The limits the authors list
  • Once the strips are folded into the model, you cannot easily serve requests for different tasks in one batch.
  • They do not expect a very thin add-on to work for every task, for example one in a different language from the original training.
Words the paper uses
  • Low-rank: a big grid of numbers that can be rebuilt from two thin ones.
  • Rank (r): how thin. Rank 1 is the thinnest possible.
  • Adaptation: customising a trained model for a task.
  • Checkpoint: a saved copy of a model.
  • Inference latency: the delay when the model answers.

Next paper: Chain-of-Thought