LoRA: Low-Rank Adaptation of Large Language Models
- Publication
- ICLR 2022Conference
- First submitted to arXiv
- arXiv:2106.09685
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{modeltuninglora2021,
title = {{LoRA: Low-Rank Adaptation of Large Language Models}},
author = {Edward J. Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen},
year = {2021},
eprint = {2106.09685},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2106.09685}
}
We have to admit it: at the scale of GPT-3 175B in 2021, most people could not train an LLM like that.
The world is large, but the companies able to train and iterate on a model of that size could be counted on one hand.
Researchers urgently wanted to use these huge models to solve all kinds of problems, so finding an efficient way to fine-tune them became important.
Defining the Problem
If you aim to fine-tune an entire model, the process would typically involve the following steps:
-
Initialize the model with pretrained weights .
-
Update the model through gradient optimization to , maximizing the conditional language model objective:
Each downstream task requires learning a different set of parameters , with the size of equal to that of the original model . For large models like GPT-3, which contains 175 billion parameters, storing and deploying multiple fine-tuned versions becomes a daunting task.
In previous research, the mainstream approach to address this issue was Adapters, which we have previously discussed:
However, the exploration does not stop there. Here are a few additional studies that are worth a look:
- [20.04] Exploring Versatile Generative Language Models with Parameter-Efficient Transfer Learning
- [20.05] AdapterFusion: Non-Destructive Task Composition for Transfer Learning
- [21.04] The Power of Scale for Parameter-Efficient Prompt Tuning
However, these methods still face certain limitations, particularly in large-scale, latency-sensitive production environments. They increase the parameter count and may interfere with parallel computation during model deployment. For example, adding an Adapter to GPT-2 results in a noticeable increase in latency, as shown in the following table:
Nonlinear adapter layers add sequential work, so few additional FLOPs can still increase latency. Parameter count depends on bottleneck width; some settings are below 1%, but this is not universal. The table measures a single GPT-2 Medium forward pass on a Quadro RTX8000, not a hardware-independent rule.
In theory, this should not significantly increase the computational load (measured in FLOPs), right?
Unfortunately, it still does.
This is because large neural networks rely heavily on parallel hardware to maintain low latency. However, Adapters require sequential processing, which can lead to significant delays during online inference, where batch sizes are typically set to 1.
What About Prefix-Tuning?
Another potential solution is Prefix-Tuning:
Prefix-Tuning represents a different strategy for adaptation. However, it has some issues: parameter performance tends to fluctuate non-monotonically during optimization, making the process challenging. Additionally, Prefix-Tuning reserves part of the input sequence length for the prefix, which reduces the available length for downstream tasks.
This makes things even worse.
To address these challenges, the authors propose low-rank decomposition of parameters. Task-specific parameter updates are encoded as a much smaller parameter set , such that:
The optimization objective is redefined to focus on this smaller set :
Solving the Problem
Before diving into the methodology, let’s first review some high-school math concepts.
Rank, Full Rank, and Low Rank
In mathematics, a matrix is a table of numbers arranged in rows and columns. For example:
Matrices play a crucial role in data processing and computation, and one of the key concepts associated with matrices is rank.
-
What is Rank?
Rank refers to the number of independent rows or columns in a matrix.
Independence means that a row (or column) cannot be represented as a linear combination of other rows (or columns). If some rows or columns can be expressed as combinations of others, they are not independent. In other words, the rank of a matrix tells us how much useful, independent information it contains.
For a matrix of size (with rows and columns), its rank is the largest number of independent rows or columns. The rank cannot exceed the smaller dimension of the matrix:
-
Full-Rank Matrices
A matrix is called full-rank when its rank equals the smaller of the number of rows or columns:
If , full rank means all rows are linearly independent; if , all columns are independent. Both hold for square full-rank matrices. This concerns linear dependence, not semantic usefulness. For example:
This is a identity matrix, where both rows are independent. The rank of this matrix is 2 (equal to ), making it a full-rank matrix. Each row or column provides new information that cannot be expressed using other rows or columns.
-
Low-Rank Matrices
A matrix is rank-deficient when its rank is below : some rows or columns can be expressed as combinations of others. In low-rank adaptation, is usually much smaller still. Storage savings come from keeping low-rank factors, not simply from a densely stored matrix having low mathematical rank.
For example:
Here, the second row is simply the first row multiplied by 2. Therefore, the matrix’s rank is 1 (only one independent row), not 2. This is a low-rank matrix. Since the rank is low, we can use a smaller matrix to approximate or describe it.
Why Are Low-Rank Matrices Useful?
In many applications, we encounter large matrices that contain redundant information. In such cases, we can use low-rank matrices to approximate the original matrix and achieve several benefits:
- Reduced Computational Cost: Computing with low-rank factors can require fewer operations and speed up algorithms.
- Lower Storage Requirements: Keeping the factors can use less memory, which is useful for large datasets.
- Dimensionality Reduction: High-dimensional data can be projected onto a lower-dimensional space while retaining essential information (e.g., PCA).
- Capacity control: A low-rank constraint can regularize, but can also underfit; improved generalization is not guaranteed.
SVD itself is an exact factorization. Keeping only leading singular values and vectors gives truncated SVD, a low-rank approximation useful in areas such as image compression and recommender systems. LoRA directly trains update factors; it does not first fully fine-tune and then run SVD.
Low-Rank Decomposition: LoRA
In neural networks, dense layers rely on matrix multiplication, where the weight matrices are typically full-rank. Previous research has shown that language-model fine-tuning can remain effective even when updates are constrained to a small random subspace, suggesting low intrinsic dimensionality.
Based on this, the authors hypothesize that weight updates during fine-tuning also exhibit low intrinsic rank. In other words, for a pretrained weight matrix , the update can be expressed as a low-rank decomposition:
where and , with .
During training, the pretrained weight matrix remains frozen, meaning its gradients are not updated. Only and are trainable parameters. These matrices are multiplied by the input vector , and the results are summed element-wise:
Matrix is initialized with values from a Gaussian distribution, while is initialized with zeros. Thus, at the start of training, is zero.
For stability, the update is scaled by:
The coefficient is held fixed when varying . The paper sets it to the first rank tried, rather than enforcing for every experiment. The learning-rate analogy depends on Adam and appropriately scaled initialization.
With sufficient rank on all relevant matrices and training of biases and remaining parameters, the update can recover the expressiveness of full fine-tuning. Optimization, convergence, and final quality require separate evaluation; adapting only is not full-model fine-tuning.
LoRA on Transformer Models
LoRA can be applied to any neural network weight matrix. In Transformer architectures, it primarily targets the self-attention module matrices:
- Four key matrices: , , , and .
- Each matrix has dimensions , even though their outputs are often split across multiple attention heads.
In the paper, the authors only fine-tune the attention weights, while freezing the MLP modules to reduce the parameter count.
This approach provides several benefits:
-
Reduced Memory and Storage Requirements:
- For large Transformer models (e.g., GPT-3 with 175B parameters), VRAM usage decreases from 1.2TB to 350GB (if ).
- If and only the query and value projection matrices are fine-tuned, the checkpoint size shrinks from 350GB to 35MB (a 10,000x reduction).
- The 35MB contains only the task update. Deployment still needs the approximately 350GB FP16 backbone. The quoted training memory is the reported aggregate requirement, not capacity that fits on one consumer GPU.
-
Lower GPU Utilization and Avoiding I/O Bottlenecks: With smaller storage needs, fewer GPUs are required for training.
-
Faster Task Switching: Tasks can be switched simply by swapping LoRA weights without reloading the entire model.
-
Training efficiency: The text reports about 25% speedup, while its footnote gives 32.5 versus 43.1 tokens/s per V100. Those numbers imply about 32.6% greater throughput or 24.6% less time for a fixed token count; these are different measures.
However, there are some drawbacks. If the matrices and are merged back into the original weight matrix to eliminate inference delays, handling batch inputs across different tasks becomes challenging. In latency-insensitive scenarios, one can avoid merging the weights and dynamically select the appropriate LoRA module for each task.
Discussion
Experimental Baselines
The authors conducted experiments on RoBERTa, DeBERTa, GPT-2, and GPT-3, using the following baselines:
-
Fine-Tuning (FT)
- Full-Model Fine-Tuning: All parameters, initialized from the pretrained weights and biases, are updated during training.
- Partial-Layer Fine-Tuning (FTTop2): Only the last two layers of GPT-2 are updated (Li & Liang, 2021).
- Bias Fine-Tuning (BitFit): Only bias vectors are trainable, while all other parameters remain frozen (Zaken et al., 2021).
-
Prefix-Embedding Tuning (PreEmbed)
Special tokens are added before or after the task input, and their embeddings are trainable.
- Strategies:
- Prefix: Special tokens are added before the task input.
- Infix: Special tokens are added after the task input.
- Parameter Count: where and are the numbers of prefix and infix tokens, respectively.
- Strategies:
-
Prefix-Layer Tuning (PreLayer)
Instead of tuning embeddings, the activation values for each Transformer layer are learned.
- Parameter Count: where is the number of Transformer layers.
-
Adapter Tuning
- AdapterH: The Houlsby design has two adapters per block, after attention and after the MLP, each before its corresponding residual addition.
- AdapterL: Applied only after the MLP module and LayerNorm (Lin et al., 2020).
- AdapterP: Similar to AdapterL (Pfeiffer et al., 2021).
- AdapterD: Some Adapters are removed for efficiency (Rücklé et al., 2020).
- Parameter Count: where is the number of Adapter layers and is the number of trainable LayerNorms.
-
LoRA Tuning
LoRA introduces low-rank matrices alongside the original weight matrices, which are updated in parallel. In most experiments, LoRA is applied only to the query matrix and value matrix .
- Parameter Count: where is the number of matrices with LoRA applied, and is the rank.
Comparison with Other Fine-Tuning Methods
The authors evaluated RoBERTa Base (125M) and RoBERTa Large (355M) models from HuggingFace for their experiments.
Rows marked † use the following two adjustments for comparison with adapters; they do not apply to every row in the table:
- Same Batch Size and Sequence Length: All tasks used the same batch size, with the sequence length set to 128 to match the Adapter baseline.
- Model Initialization: For MRPC, RTE, and STS-B tasks, the models were initialized from pretrained weights, not from MNLI-adapted models (as in other fine-tuning baselines).
The results show that LoRA performs similarly to full-model fine-tuning on both RoBERTa Base and RoBERTa Large. In most cases, LoRA outperforms both Adapters and other parameter-efficient tuning methods.
DeBERTa is a more advanced variant of BERT with larger training scales and competitive performance on benchmarks like GLUE and SuperGLUE. The authors also tested LoRA on DeBERTa, with the results (shown in the table above) indicating that LoRA often outperforms full-model fine-tuning.
Fine-Tuning GPT-3
Finally, we have reached the GPT-3 fine-tuning results! On WikiSQL, MultiNLI-matched and SAMSum in the reported settings, LoRA performs better overall than the tested Adapters and other efficient tuning methods, while matching or exceeding full fine-tuning.
However, increasing the number of trainable parameters does not always lead to improved performance, as seen in the following chart:
With prefix-embedding tuning, performance drops when using more than 256 special tokens. Similarly, prefix-layer tuning shows degraded performance beyond 32 special tokens. This decline could be due to the input distribution diverging from the pretrained data distribution when too many special tokens are used.
Which Weight Matrices Should LoRA Target in Transformers?
Given a limited parameter budget, which weight matrices should be adapted to achieve optimal performance in downstream tasks?
The authors conducted experiments on GPT-3 (175B), setting the number of trainable parameters to 18 million (approximately 35MB with FP16 precision). Across all 96 layers, if LoRA is applied to one type of attention weight, the rank is set to . If two types are adapted, the rank is reduced to .
These are validation accuracies on WikiSQL and MultiNLI. The results in the table suggest:
- Adapting only or leads to a significant drop in performance.
- Adapting gives 73.7% on WikiSQL and 91.3% on MultiNLI. Adapting all four matrix types at gives 73.7% and 91.7%. Table 5 therefore supports distributing adaptation within this budget, not a unique or universal Q/V optimum.
Even with a small rank , can capture sufficient information. This suggests that adapting more types of weight matrices with a smaller rank is more effective than adapting only one type with a higher rank.
What is the Optimal Rank for LoRA?
LoRA achieves excellent performance even with very small ranks, especially when applied to .
Small ranks are competitive in these GPT-3 experiments on WikiSQL and MultiNLI, and increasing rank does not consistently improve scores. The authors explicitly caution that other languages, tasks or datasets may need higher ranks; is not a universal optimum.
Subspace Similarity Analysis
The authors also explored the similarity between subspaces for different values of and across different random seeds. They performed Singular Value Decomposition (SVD) on these matrices to obtain the right singular matrices:
Their goal was to determine how much of the subspace formed by the top singular vectors of overlaps with the subspace formed by the top singular vectors of .
They measured the similarity using normalized subspace similarity based on Grassmann distance:
Here, represents the first columns of the singular matrix .
Here the paper calls the right-singular-vector matrix . In the usual SVD notation , these vectors are columns of . For , the compared bases lie in the same -dimensional input space.
The resulting comparison is shown below:
The directions of the top singular vectors overlap significantly between and , while the overlap is smaller for other directions. Overall, the subspaces of and for (or ) share a dimension with similarity .
This may help explain why rank works well on these downstream tasks: useful information may be concentrated in the top singular-vector directions.
Let’s summarize the key takeaways from the paper:
- Low-Rank Adaptation Matrices Can Be Sufficient: On the tasks tested here, LoRA remains comparable to full-model fine-tuning even with very small ranks.
- Adapting More Types of Weight Matrices Can Be More Effective than Increasing Rank: Under Table 5's tasks and parameter budget, adapting more types (e.g., both and ) with a smaller rank works better than focusing on one type with a higher rank.
Conclusion
Compared with other fine-tuning methods, LoRA gives us a lot!
In the GPT-3 experiment, it cuts trainable task-specific parameters by roughly 10,000 times while keeping performance comparable in the tested tasks. That saving applies to the task update and its checkpoint; the large frozen backbone and substantial training memory are still needed.
- At last, more of us can fine-tune an LLM!
In 2021, adapting a shared large model to many applications became more achievable, even if a 175B model still required serious hardware. It is exciting to think how many more settings could benefit from such models.
The next decade should be lively. Let's see what other interesting research appears!