Back to paper library
arXiv 2021ICLR 2022Model adaptationPaper notes

LoRA

One in Ten Thousand LLM

Original paper

LoRA: Low-Rank Adaptation of Large Language Models

Paper authors

Edward J. Hu · Yelong Shen · Phillip Wallis …

Show all 8 authors

Edward J. Hu · Yelong Shen · Phillip Wallis · Zeyuan Allen-Zhu · Yuanzhi Li · Shean Wang · Lu Wang · Weizhu Chen

Publication
ICLR 2022Conference
First submitted to arXiv
arXiv:2106.09685
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{modeltuninglora2021,
  title = {{LoRA: Low-Rank Adaptation of Large Language Models}},
  author = {Edward J. Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen},
  year = {2021},
  eprint = {2106.09685},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.09685}
}
On this page

We have to admit it: at the scale of GPT-3 175B in 2021, most people could not train an LLM like that.

The world is large, but the companies able to train and iterate on a model of that size could be counted on one hand.

Researchers urgently wanted to use these huge models to solve all kinds of problems, so finding an efficient way to fine-tune them became important.

Defining the Problem

If you aim to fine-tune an entire model, the process would typically involve the following steps:

  1. Initialize the model with pretrained weights Φ0\Phi_0.

  2. Update the model through gradient optimization to Φ0+ΔΦ\Phi_0 + \Delta \Phi, maximizing the conditional language model objective:

    max⁡Φ∑(x,y)∈Z∑t=1∣y∣log⁡PΦ(yt∣x,y<t)\max_\Phi \sum_{(x,y) \in Z} \sum_{t=1}^{|y|} \log P_\Phi(y_t | x, y_{<t})

Each downstream task requires learning a different set of parameters ΔΦ\Delta \Phi, with the size of ∣ΔΦ∣|\Delta \Phi| equal to that of the original model ∣Φ0∣|\Phi_0|. For large models like GPT-3, which contains 175 billion parameters, storing and deploying multiple fine-tuned versions becomes a daunting task.

In previous research, the mainstream approach to address this issue was Adapters, which we have previously discussed:

However, the exploration does not stop there. Here are a few additional studies that are worth a look:

However, these methods still face certain limitations, particularly in large-scale, latency-sensitive production environments. They increase the parameter count and may interfere with parallel computation during model deployment. For example, adding an Adapter to GPT-2 results in a noticeable increase in latency, as shown in the following table:

Nonlinear adapter layers add sequential work, so few additional FLOPs can still increase latency. Parameter count depends on bottleneck width; some settings are below 1%, but this is not universal. The table measures a single GPT-2 Medium forward pass on a Quadro RTX8000, not a hardware-independent rule.

In theory, this should not significantly increase the computational load (measured in FLOPs), right?

Unfortunately, it still does.

This is because large neural networks rely heavily on parallel hardware to maintain low latency. However, Adapters require sequential processing, which can lead to significant delays during online inference, where batch sizes are typically set to 1.

What About Prefix-Tuning?

Another potential solution is Prefix-Tuning:

Prefix-Tuning represents a different strategy for adaptation. However, it has some issues: parameter performance tends to fluctuate non-monotonically during optimization, making the process challenging. Additionally, Prefix-Tuning reserves part of the input sequence length for the prefix, which reduces the available length for downstream tasks.

This makes things even worse.

To address these challenges, the authors propose low-rank decomposition of parameters. Task-specific parameter updates ΔΦ\Delta \Phi are encoded as a much smaller parameter set Θ\Theta, such that:

∣Θ∣≪∣Φ0∣|\Theta| \ll |\Phi_0|

The optimization objective is redefined to focus on this smaller set Θ\Theta:

max⁡Θ∑(x,y)∈Z∑t=1∣y∣log⁡PΦ0+ΔΦ(Θ)(yt∣x,y<t)\max_\Theta \sum_{(x,y) \in Z} \sum_{t=1}^{|y|} \log P_{\Phi_0 + \Delta \Phi(\Theta)}(y_t | x, y_{<t})

Solving the Problem

Before diving into the methodology, let’s first review some high-school math concepts.

Rank, Full Rank, and Low Rank

In mathematics, a matrix is a table of numbers arranged in rows and columns. For example:

A=[123456789]\mathbf{A} = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{bmatrix}

Matrices play a crucial role in data processing and computation, and one of the key concepts associated with matrices is rank.


  1. What is Rank?

    Rank refers to the number of independent rows or columns in a matrix.

    Independence means that a row (or column) cannot be represented as a linear combination of other rows (or columns). If some rows or columns can be expressed as combinations of others, they are not independent. In other words, the rank of a matrix tells us how much useful, independent information it contains.

    For a matrix AA of size m×nm \times n (with mm rows and nn columns), its rank is the largest number of independent rows or columns. The rank cannot exceed the smaller dimension of the matrix:

    rank(A)≤min⁡(m,n)\text{rank}(A) \leq \min(m, n)

  1. Full-Rank Matrices

    A matrix is called full-rank when its rank equals the smaller of the number of rows or columns:

    rank(A)=min⁡(m,n)\text{rank}(A) = \min(m, n)

    If m≤nm\le n, full rank means all rows are linearly independent; if n≤mn\le m, all columns are independent. Both hold for square full-rank matrices. This concerns linear dependence, not semantic usefulness. For example:

    I=[1001]\mathbf{I} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}

    This is a 2×22 \times 2 identity matrix, where both rows are independent. The rank of this matrix is 2 (equal to min⁡(2,2)\min(2, 2)), making it a full-rank matrix. Each row or column provides new information that cannot be expressed using other rows or columns.


  1. Low-Rank Matrices

    A matrix is rank-deficient when its rank is below min⁡(m,n)\min(m,n): some rows or columns can be expressed as combinations of others. In low-rank adaptation, rr is usually much smaller still. Storage savings come from keeping low-rank factors, not simply from a densely stored matrix having low mathematical rank.

    For example:

    B=[1224]\mathbf{B} = \begin{bmatrix} 1 & 2 \\ 2 & 4 \end{bmatrix}

    Here, the second row is simply the first row multiplied by 2. Therefore, the matrix’s rank is 1 (only one independent row), not 2. This is a low-rank matrix. Since the rank is low, we can use a smaller matrix to approximate or describe it.

Why Are Low-Rank Matrices Useful?

In many applications, we encounter large matrices that contain redundant information. In such cases, we can use low-rank matrices to approximate the original matrix and achieve several benefits:

  1. Reduced Computational Cost: Computing with low-rank factors can require fewer operations and speed up algorithms.
  2. Lower Storage Requirements: Keeping the factors can use less memory, which is useful for large datasets.
  3. Dimensionality Reduction: High-dimensional data can be projected onto a lower-dimensional space while retaining essential information (e.g., PCA).
  4. Capacity control: A low-rank constraint can regularize, but can also underfit; improved generalization is not guaranteed.

SVD itself is an exact factorization. Keeping only leading singular values and vectors gives truncated SVD, a low-rank approximation useful in areas such as image compression and recommender systems. LoRA directly trains update factors; it does not first fully fine-tune and then run SVD.

Low-Rank Decomposition: LoRA

In neural networks, dense layers rely on matrix multiplication, where the weight matrices are typically full-rank. Previous research has shown that language-model fine-tuning can remain effective even when updates are constrained to a small random subspace, suggesting low intrinsic dimensionality.

Based on this, the authors hypothesize that weight updates during fine-tuning also exhibit low intrinsic rank. In other words, for a pretrained weight matrix W0∈Rd×kW_0 \in \mathbb{R}^{d \times k}, the update can be expressed as a low-rank decomposition:

W0+ΔW=W0+BAW_0 + \Delta W = W_0 + BA

where B∈Rd×rB \in \mathbb{R}^{d \times r} and A∈Rr×kA \in \mathbb{R}^{r \times k}, with r≪min⁡(d,k)r \ll \min(d, k).

During training, the pretrained weight matrix W0W_0 remains frozen, meaning its gradients are not updated. Only AA and BB are trainable parameters. These matrices are multiplied by the input vector xx, and the results are summed element-wise:

h=W0x+ΔWx=W0x+BAxh = W_0 x + \Delta W x = W_0 x + BA x

Matrix AA is initialized with values from a Gaussian distribution, while BB is initialized with zeros. Thus, at the start of training, ΔW=BA\Delta W = BA is zero.

For stability, the update ΔWx\Delta W x is scaled by:

αr⋅ΔWx\frac{\alpha}{r} \cdot \Delta W x

The coefficient α\alpha is held fixed when varying rr. The paper sets it to the first rank tried, rather than enforcing α=r\alpha=r for every experiment. The learning-rate analogy depends on Adam and appropriately scaled initialization.

With sufficient rank on all relevant matrices and training of biases and remaining parameters, the update can recover the expressiveness of full fine-tuning. Optimization, convergence, and final quality require separate evaluation; adapting only Wq,WvW_q,W_v is not full-model fine-tuning.

LoRA on Transformer Models

LoRA can be applied to any neural network weight matrix. In Transformer architectures, it primarily targets the self-attention module matrices:

  • Four key matrices: WqW_q, WkW_k, WvW_v, and WoW_o.
  • Each matrix has dimensions dmodel×dmodeld_{\text{model}} \times d_{\text{model}}, even though their outputs are often split across multiple attention heads.

In the paper, the authors only fine-tune the attention weights, while freezing the MLP modules to reduce the parameter count.

This approach provides several benefits:

  • Reduced Memory and Storage Requirements:

    • For large Transformer models (e.g., GPT-3 with 175B parameters), VRAM usage decreases from 1.2TB to 350GB (if r≪dmodelr \ll d_{\text{model}}).
    • If r=4r = 4 and only the query and value projection matrices are fine-tuned, the checkpoint size shrinks from 350GB to 35MB (a 10,000x reduction).
    • The 35MB contains only the task update. Deployment still needs the approximately 350GB FP16 backbone. The quoted training memory is the reported aggregate requirement, not capacity that fits on one consumer GPU.
  • Lower GPU Utilization and Avoiding I/O Bottlenecks: With smaller storage needs, fewer GPUs are required for training.

  • Faster Task Switching: Tasks can be switched simply by swapping LoRA weights without reloading the entire model.

  • Training efficiency: The text reports about 25% speedup, while its footnote gives 32.5 versus 43.1 tokens/s per V100. Those numbers imply about 32.6% greater throughput or 24.6% less time for a fixed token count; these are different measures.

However, there are some drawbacks. If the matrices AA and BB are merged back into the original weight matrix WW to eliminate inference delays, handling batch inputs across different tasks becomes challenging. In latency-insensitive scenarios, one can avoid merging the weights and dynamically select the appropriate LoRA module for each task.

Discussion

Experimental Baselines

The authors conducted experiments on RoBERTa, DeBERTa, GPT-2, and GPT-3, using the following baselines:

  1. Fine-Tuning (FT)

    • Full-Model Fine-Tuning: All parameters, initialized from the pretrained weights and biases, are updated during training.
    • Partial-Layer Fine-Tuning (FTTop2): Only the last two layers of GPT-2 are updated (Li & Liang, 2021).
    • Bias Fine-Tuning (BitFit): Only bias vectors are trainable, while all other parameters remain frozen (Zaken et al., 2021).

  1. Prefix-Embedding Tuning (PreEmbed)

    Special tokens are added before or after the task input, and their embeddings are trainable.

    • Strategies:
      • Prefix: Special tokens are added before the task input.
      • Infix: Special tokens are added after the task input.
    • Parameter Count: ∣Θ∣=dmodel×(lp+li)|\Theta| = d_{\text{model}} \times (l_p + l_i) where lpl_p and lil_i are the numbers of prefix and infix tokens, respectively.

  1. Prefix-Layer Tuning (PreLayer)

    Instead of tuning embeddings, the activation values for each Transformer layer are learned.

    • Parameter Count: ∣Θ∣=L×dmodel×(lp+li)|\Theta| = L \times d_{\text{model}} \times (l_p + l_i) where LL is the number of Transformer layers.

  1. Adapter Tuning

    • AdapterH: The Houlsby design has two adapters per block, after attention and after the MLP, each before its corresponding residual addition.
    • AdapterL: Applied only after the MLP module and LayerNorm (Lin et al., 2020).
    • AdapterP: Similar to AdapterL (Pfeiffer et al., 2021).
    • AdapterD: Some Adapters are removed for efficiency (Rücklé et al., 2020).
    • Parameter Count: ∣Θ∣=L^Adpt×(2×dmodel×r+r+dmodel)+2×L^LN×dmodel|\Theta| = \hat{L}_{\text{Adpt}} \times (2 \times d_{\text{model}} \times r + r + d_{\text{model}}) + 2 \times \hat{L}_{\text{LN}} \times d_{\text{model}} where L^Adpt\hat{L}_{\text{Adpt}} is the number of Adapter layers and L^LN\hat{L}_{\text{LN}} is the number of trainable LayerNorms.

  1. LoRA Tuning

    LoRA introduces low-rank matrices alongside the original weight matrices, which are updated in parallel. In most experiments, LoRA is applied only to the query matrix WqW_q and value matrix WvW_v.

    • Parameter Count: ∣Θ∣=2×L^LoRA×dmodel×r|\Theta| = 2 \times \hat{L}_{\text{LoRA}} \times d_{\text{model}} \times r where L^LoRA\hat{L}_{\text{LoRA}} is the number of matrices with LoRA applied, and rr is the rank.

Comparison with Other Fine-Tuning Methods

The authors evaluated RoBERTa Base (125M) and RoBERTa Large (355M) models from HuggingFace for their experiments.

Rows marked † use the following two adjustments for comparison with adapters; they do not apply to every row in the table:

  1. Same Batch Size and Sequence Length: All tasks used the same batch size, with the sequence length set to 128 to match the Adapter baseline.
  2. Model Initialization: For MRPC, RTE, and STS-B tasks, the models were initialized from pretrained weights, not from MNLI-adapted models (as in other fine-tuning baselines).

The results show that LoRA performs similarly to full-model fine-tuning on both RoBERTa Base and RoBERTa Large. In most cases, LoRA outperforms both Adapters and other parameter-efficient tuning methods.

DeBERTa is a more advanced variant of BERT with larger training scales and competitive performance on benchmarks like GLUE and SuperGLUE. The authors also tested LoRA on DeBERTa, with the results (shown in the table above) indicating that LoRA often outperforms full-model fine-tuning.

Fine-Tuning GPT-3

Finally, we have reached the GPT-3 fine-tuning results! On WikiSQL, MultiNLI-matched and SAMSum in the reported settings, LoRA performs better overall than the tested Adapters and other efficient tuning methods, while matching or exceeding full fine-tuning.

However, increasing the number of trainable parameters does not always lead to improved performance, as seen in the following chart:

With prefix-embedding tuning, performance drops when using more than 256 special tokens. Similarly, prefix-layer tuning shows degraded performance beyond 32 special tokens. This decline could be due to the input distribution diverging from the pretrained data distribution when too many special tokens are used.

Which Weight Matrices Should LoRA Target in Transformers?

Given a limited parameter budget, which weight matrices should be adapted to achieve optimal performance in downstream tasks?

The authors conducted experiments on GPT-3 (175B), setting the number of trainable parameters to 18 million (approximately 35MB with FP16 precision). Across all 96 layers, if LoRA is applied to one type of attention weight, the rank is set to r=8r = 8. If two types are adapted, the rank is reduced to r=4r = 4.

These are validation accuracies on WikiSQL and MultiNLI. The results in the table suggest:

  • Adapting only ΔWq\Delta W_q or ΔWk\Delta W_k leads to a significant drop in performance.
  • Adapting Wq,WvW_q,W_v gives 73.7% on WikiSQL and 91.3% on MultiNLI. Adapting all four matrix types at r=2r=2 gives 73.7% and 91.7%. Table 5 therefore supports distributing adaptation within this budget, not a unique or universal Q/V optimum.

Even with a small rank r=4r = 4, ΔW\Delta W can capture sufficient information. This suggests that adapting more types of weight matrices with a smaller rank is more effective than adapting only one type with a higher rank.

What is the Optimal Rank rr for LoRA?

LoRA achieves excellent performance even with very small ranks, especially when applied to {Wq,Wv}\{W_q, W_v\}.

Small ranks are competitive in these GPT-3 experiments on WikiSQL and MultiNLI, and increasing rank does not consistently improve scores. The authors explicitly caution that other languages, tasks or datasets may need higher ranks; r=1r=1 is not a universal optimum.

Subspace Similarity Analysis

The authors also explored the similarity between subspaces for different values of rr and across different random seeds. They performed Singular Value Decomposition (SVD) on these matrices to obtain the right singular matrices:

UAr=8andUAr=64U_{A_{r=8}} \quad \text{and} \quad U_{A_{r=64}}

Their goal was to determine how much of the subspace formed by the top ii singular vectors of UAr=8U_{A_{r=8}} overlaps with the subspace formed by the top jj singular vectors of UAr=64U_{A_{r=64}}.

They measured the similarity using normalized subspace similarity based on Grassmann distance:

ϕ(Ar=8,Ar=64,i,j)=∥(UAr=8i)⊤UAr=64j∥F2min⁡(i,j)∈[0,1]\phi(A_{r=8}, A_{r=64}, i, j) = \frac{\left\| \left(U_{A_{r=8}}^{i}\right)^\top U_{A_{r=64}}^{j} \right\|_F^2}{\min(i, j)} \in [0, 1]

Here, UAr=8iU_{A_{r=8}}^{i} represents the first ii columns of the singular matrix UAr=8U_{A_{r=8}}.

Here the paper calls the right-singular-vector matrix UAU_A. In the usual SVD notation A=UΣV⊤A=U\Sigma V^\top, these vectors are columns of VV. For A∈Rr×kA\in\mathbb R^{r\times k}, the compared bases lie in the same kk-dimensional input space.

The resulting comparison is shown below:

The directions of the top singular vectors overlap significantly between Ar=8A_{r=8} and Ar=64A_{r=64}, while the overlap is smaller for other directions. Overall, the subspaces of Ar=8A_{r=8} and Ar=64A_{r=64} for ΔWv\Delta W_v (or ΔWq\Delta W_q) share a dimension with similarity ϕ>0.5\phi > 0.5.

This may help explain why rank r=1r = 1 works well on these downstream tasks: useful information may be concentrated in the top singular-vector directions.


Let’s summarize the key takeaways from the paper:

  1. Low-Rank Adaptation Matrices Can Be Sufficient: On the tasks tested here, LoRA remains comparable to full-model fine-tuning even with very small ranks.
  2. Adapting More Types of Weight Matrices Can Be More Effective than Increasing Rank: Under Table 5's tasks and parameter budget, adapting more types (e.g., both WqW_q and WvW_v) with a smaller rank works better than focusing on one type with a higher rank.

Conclusion

Compared with other fine-tuning methods, LoRA gives us a lot!

In the GPT-3 experiment, it cuts trainable task-specific parameters by roughly 10,000 times while keeping performance comparable in the tested tasks. That saving applies to the task update and its checkpoint; the large frozen backbone and substantial training memory are still needed.

  • At last, more of us can fine-tune an LLM!

In 2021, adapting a shared large model to many applications became more achievable, even if a 175B model still required serious hardware. It is exciting to think how many more settings could benefit from such models.

The next decade should be lively. Let's see what other interesting research appears!