GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
- Publication
- ICML 2024Conference
- First submitted to arXiv
- arXiv:2403.03507
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{optimizersgalore2024,
title = {{GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection}},
author = {Jiawei Zhao and Zhenyu Zhang and Beidi Chen and Zhangyang Wang and Anima Anandkumar and Yuandong Tian},
year = {2024},
eprint = {2403.03507},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2403.03507}
}
How much memory does it take to train a 7-billion-parameter LLaMA from scratch with a batch of just one sequence of length 2,048?
The paper works it out right at the start. BF16 weights take 14 GB; weight gradients take 14 GB, plus 14 GB each for Adam's first and second moments, 42 GB in all; activations take another 2 GB. That's 58 GB in total. An RTX 4090 has only 24 GB.
This paper's claim is that with its method, combined with 8-bit Adam and per-layer weight updates, 7B pre-training measures in at just 22 GB. No model splitting, no activation checkpointing, no offloading to the CPU, and it fits on a single 4090.
Defining the problem
There are plenty of established ways to save memory. Activation checkpointing trades recomputation for memory, ZeRO shards optimizer state across GPUs or offloads it to the CPU, 8-bit Adam quantizes the moments, and Adafactor factors the second moment into row and column sums. Each saves a share, at the cost of time, communication or stability.
Another route is LoRA. It writes the weights as , freezes , and trains only two small matrices and , so the gradients and optimizer state shrink to size .
LoRA has been very successful for fine-tuning, but it struggles with pre-training. Starting from scratch, is just a random initialization, and everything the model can learn is locked into dimensions; refinements like ReLoRA still need a stretch of full-rank warmup training first. The authors see two reasons. The optimal weights aren't necessarily low-rank, and the reparameterization itself changes the training dynamics. To take an extreme example, if the optimal solution of a linear regression is high-rank, then once is restricted to low rank, no optimizer can reach it.
So the authors asked a different question. The weights needn't be low-rank, but what about the gradients?
Solving the problem
Gradients become low-rank
The authors' theoretical starting point is a class of "reversible networks," which includes layers like linear layers (without bias), ReLU and leaky ReLU, along with their compositions and sums, residual networks included. For such networks, the weight gradient over a batch can be written as
Here and are positive semidefinite matrices. Lemma 3.3 proves that if , and stay fixed after some point in time, then under plain SGD updates, the stable rank of (, which you can think of as the "effective rank") falls exponentially, until only the part determined by the smallest eigensubspace remains. Corollary 3.4 gives a more concrete upper bound. When the gradient is a sum of outer products from data points and the input vectors have rank , the final stable rank is at most ; and a sum of outer products itself has rank at most , from which the paper concludes that in the long run the stable rank is at most .
Intuitively, a linear layer's gradient is a sum of outer products of "each data point's backpropagated error × each data point's input," so it only has a limited number of directions to begin with; after training for a while, most of its mass concentrates in just a few of them.
This theory covers only reversible networks, fixed coefficient matrices and SGD updates. For Transformers, the authors use a different framework in the appendix to prove only that the gradient of the feed-forward layer's first projection becomes low-rank; self-attention and Adam both lie outside the scope of the proofs.
Running Adam in a small projected space
Since the gradient concentrates in a few directions, keep optimizer state only along those directions. GaLore (Gradient Low-Rank Projection) is defined as
where is any stateful optimizer, such as Adam. In practice, only a one-sided projection is used. For an weight with , an SVD of the current gradient is taken every steps, and the top left singular vectors become the projection matrix,
Then at every step:
(The paper defines as the negative gradient, so the update is added to the weights.) is a scale factor that plays a role like LoRA's , but doesn't change with .
Let's walk through one weight from a LLaMA 7B feed-forward layer, of shape ; the paper's 7B experiment uses . The gradient is ; the projection matrix is ; the projected is only , and so are Adam's and . Adam would originally store million numbers; now and together take 22.54 million, plus 4.19 million for , about 26.74 million in all, roughly 30% of the original. The update itself is still a full matrix, and the weights stay full-rank.
Overall, GaLore stores numbers for an weight, while LoRA stores (the original weights plus two adapters, plus the adapters' two moments).
There's one more difference from LoRA. With at full rank, , and with plain SGD, GaLore's trajectory is identical to ordinary training; for LoRA, even if is full-rank, optimizing and together follows a different trajectory.
The authors also prove convergence under a fixed projection (Theorem 3.8). The condition is , that is, stateless SGD, and it requires the smallest eigenvalue of the projected subproblem to exceed a threshold set by continuity constants. The theorem suggests aiming the projection at the largest eigensubspace, which is the rationale for taking the top directions of the gradient's SVD. The authors also state explicitly that convergence only reflects good progress within this subspace, with no guarantee of overall performance.
Switching subspaces
Moving only within one fixed subspace, the change in the weights would never exceed dimensions, which brings back LoRA's old problem. GaLore switches subspaces every steps:
Each segment is low-rank, but added up they can be full-rank.

The blue plane is the subspace for the first steps, and the weights travel some distance within it; at , the SVD is recomputed from the current gradient, switching to the orange plane, and the weights carry on from where they were.
How should be chosen? Switch too often, and the SVD overhead grows, the fixed projection the theorem requires is broken, and the moments Adam has accumulated get distorted because the basis keeps changing; switch too rarely, and you may get stuck in a subspace that no longer matters. The authors found little difference for anywhere between 50 and 1,000, use 200 for all pre-training, and estimate the total SVD overhead at under 10%. Also, when Algorithm 2 switches subspaces, and are carried over as they are, neither reset nor transformed into the new basis.
Two more tricks on top
GaLore compresses only the optimizer state, so the authors stack two existing tricks on top. One is replacing Adam with 8-bit Adam; "8-bit GaLore" below refers to this combination. The other is per-layer weight updates. As soon as backpropagation finishes computing a layer's gradient, that layer is updated immediately, so the whole model's gradients never have to be held until the end; this is the approach of the LOMO line of work.
Discussion
Pre-training from 60M to 1B
The experiments follow ReLoRA's setup, a LLaMA architecture (RMSNorm, SwiGLU) trained on C4 without repeating data, with sequence length 256, 131K tokens per step, and BF16 throughout. Each method at each size picks the best of five learning rates; GaLore uses learning rate 0.01, and at every size, for an effective learning rate of 0.0025, larger than the 0.001 or below that the full-rank baselines usually use.
| Method | 60M | 130M | 350M | 1B |
|---|---|---|---|---|
| Full-rank | 34.06 | 25.08 | 18.80 | 15.56 |
| GaLore | 34.88 | 25.36 | 18.95 | 15.64 |
| LoRA | 34.99 | 33.92 | 25.58 | 19.21 |
| ReLoRA | 37.04 | 29.37 | 29.08 | 18.33 |
| 78.18 | 45.51 | 37.41 | 142.53 |
Compiled from Table 2 of the paper; C4 validation perplexity, lower is better. Ranks are 128, 256, 256 and 512, respectively (model dimensions 512, 768, 1024 and 2048).
GaLore tracks full-rank closely at all four sizes, only 0.08 behind at 1B; LoRA and ReLoRA fall further behind as models grow. Factoring the weights directly into and training from scratch is the worst, with a perplexity of 142.53 at 1B. For memory, the estimated parameters plus optimizer state at 1B are 7.80 GB for full-rank, 4.38 GB for GaLore and 6.17 GB for LoRA.
Keep two things in mind when reading this table. The memory here is a BF16 size estimated from parameter counts, excluding activations and gradients. And ReLoRA is compared with its full-rank warmup removed, which the authors say is for fairness, though its original design requires that warmup; LoRA, too, was originally designed for fine-tuning.
7B and that 4090
The 7B experiment ran for 150,000 steps and 19.7 billion tokens on 8 nodes with 64 A100s in total. Because compute was limited, 8-bit GaLore () and 8-bit Adam were each run only once, without tuning.
| Method | 40K steps | 80K steps | 120K steps | 150K steps |
|---|---|---|---|---|
| 8-bit GaLore | 17.94 | 15.39 | 14.95 | 14.65 |
| 8-bit Adam | 18.09 | 15.47 | 14.83 | 14.61 |
Compiled from Table 3 of the paper; C4 validation perplexity.
GaLore is slightly better early on, 8-bit Adam pulls ahead later, and the final gap is 0.04.
That 4090 shows up in a separate experiment. The authors measured actual memory on a single device, with no activation checkpointing, no offloading, and a token batch size of 256.

Look at the bottom two bars. 8-bit Adam without retained gradients is about 31 GB; switching to 8-bit GaLore shortens the optimizer-state segment (the darker green) considerably, for about 22 GB in total, to the left of the red dashed line at 24 GB. Compared with 8-bit Adam, the low-rank projection saves 9.6 GB of optimizer state, and per-layer updates save 13.5 GB of weight gradients; compared with BF16 AdamW, the total saving is 63.3%.
So "pre-training 7B on a 4090" measures whether the memory fits. The measurement used a batch of only 256 tokens (at most about 500 without checkpointing), and even with activation checkpointing it only goes up to 4,096; the 7B experiment the paper actually ran to completion used 64 A100s. The authors envision data parallelism across many consumer GPUs, which the paper never actually ran.
Speed and hyperparameters
Saving memory costs time. In measurements on LLaMA 1B with a token batch of 256, 8-bit GaLore with per-layer updates processes 1,019.63 tokens per second against 1,205.31 for 8-bit Adam, which the authors call about 17% overhead; with per-layer updates turned off, GaLore goes back up to 1,109.38.

The left plot is the update frequency . All four ranks are U-shaped, getting worse if you switch too often or too rarely, with the bottom around one switch every 250 to 500 steps; the smaller the rank, the more often it needs to switch. The right plot is rank. Within a certain range, lowering the rank only slows convergence somewhat; rank 128 at 80,000 steps beats rank 512 at 20,000. In other words, when memory is short, you can trade time for it.
Fine-tuning
The fine-tuning comparisons are much milder than in pre-training. For RoBERTa-Base on GLUE at rank 4, GaLore averages 85.89, LoRA 85.61 and full fine-tuning 86.28; at rank 8 it's 85.94 versus 85.93, essentially a tie. The memory in the table, 253M versus 257M, again counts parameters plus optimizer state. In the appendix, for LLaMA-7B fine-tuned on OpenAssistant conversation data, test perplexity is 2.95 for GaLore and 2.94 for LoRA; the clearer gaps are BERT-Base on SQuAD (F1 88.29 versus 86.11) and Phi-2 fine-tuned on OpenAssistant (perplexity 3.83 versus 4.24).
Also, GaLore updates the full weights at every step, so what you get after training is an entire new model; LoRA lets you store just a small adapter that you can attach or detach at any time. Which is more convenient depends on your deployment needs.
Later scrutiny
The cost of SVD. The paper estimates SVD overhead at under 10%, but people who measured it see things differently. The author of LLaMA-Factory ran a third-party benchmark the day after GaLore's release. Fine-tuning LLaMA-2 7B on an A100 with activation checkpointing on, 16-bit GaLore without retained gradients needed only 18 GB against 40 GB for 8-bit AdamW, and throughput on ordinary steps held up too; but a note under the table says that each SVD on 7B takes about 10 minutes, which wasn't counted. Later, the team behind APOLLO (including two of GaLore's co-authors) also reported that a single subspace update on LLaMA-7B takes about 10 minutes, with throughput dropping every 200 steps. In the Hugging Face integration, layer-wise mode runs only on a single GPU, doesn't support DDP, and gradient clipping and DeepSpeed may not work properly.
Convergence. A paper by He et al. in October 2024 gave an explicit counterexample, proving that under standard stochastic optimization assumptions, GaLore doesn't necessarily converge to a stationary point. The reason is that SVD picks only the largest directions; late in training the true gradient gets small while the noise remains, and the top singular vectors may capture nothing but noise. GaLore has guarantees only when there's no noise, when the batch grows with the step count (for example ), or when the noise is the same in every direction. Their proposed GoLore uses random projections instead, and they recommend using GaLore early on and switching to GoLore for the final stretch.
Scale. In April 2025, two of GaLore's authors, Jiawei Zhao and Yuandong Tian, published GaLore 2 together with Meta's PyTorch team. They state that each SVD for LLaMA 7B takes up to 20 minutes, and switching to randomized SVD made it 15 times faster; they also note that SVD signs aren't unique, so the projections from two decompositions may not line up. This time they trained LLaMA 7B from scratch on 500 billion tokens across 32 nodes with 256 H100s, with 8-bit Adam as the baseline. GaLore trailed slightly for the first 150 billion tokens, pulled ahead around 200 billion, and the two ended up comparable.
Conclusion
GaLore's most fundamental difference from LoRA is that it keeps the weights full-rank, puts only the optimizer state into the gradient's low-rank subspace, and switches subspaces periodically. In pre-training up to 1B, it trails full-rank Adam by only a few tenths of perplexity, far better than LoRA; 7B was run only once, and ended up about the same as 8-bit Adam. The price is an SVD every few hundred steps, and a theory that holds only under specific conditions.
If you want full-parameter fine-tuning on a single consumer GPU, GaLore is worth a try. Before you start, measure how long one SVD takes, and check whether your existing multi-GPU setup still works once per-layer updates are on. If you just want to save memory for fine-tuning, LoRA or QLoRA is usually less hassle.
And that 4090? GaLore 2's scale validation used 256 H100s.