Back to paper library
arXiv 2024ICML 2024OptimizersPaper notes

GaLore

The 4090 dream

Original paper

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

Paper authors

Jiawei Zhao · Zhenyu Zhang · Beidi Chen · Zhangyang Wang · Anima Anandkumar · Yuandong Tian

Publication
ICML 2024Conference
First submitted to arXiv
arXiv:2403.03507
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{optimizersgalore2024,
  title = {{GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection}},
  author = {Jiawei Zhao and Zhenyu Zhang and Beidi Chen and Zhangyang Wang and Anima Anandkumar and Yuandong Tian},
  year = {2024},
  eprint = {2403.03507},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.03507}
}
On this page

How much memory does it take to train a 7-billion-parameter LLaMA from scratch with a batch of just one sequence of length 2,048?

The paper works it out right at the start. BF16 weights take 14 GB; weight gradients take 14 GB, plus 14 GB each for Adam's first and second moments, 42 GB in all; activations take another 2 GB. That's 58 GB in total. An RTX 4090 has only 24 GB.

This paper's claim is that with its method, combined with 8-bit Adam and per-layer weight updates, 7B pre-training measures in at just 22 GB. No model splitting, no activation checkpointing, no offloading to the CPU, and it fits on a single 4090.

Defining the problem

There are plenty of established ways to save memory. Activation checkpointing trades recomputation for memory, ZeRO shards optimizer state across GPUs or offloads it to the CPU, 8-bit Adam quantizes the moments, and Adafactor factors the second moment into row and column sums. Each saves a share, at the cost of time, communication or stability.

Another route is LoRA. It writes the weights as W=W0+BAW = W_0 + BA, freezes W0W_0, and trains only two small matrices B∈Rm×rB \in \mathbb{R}^{m\times r} and A∈Rr×nA \in \mathbb{R}^{r\times n}, so the gradients and optimizer state shrink to size (m+n)r(m+n)r.

LoRA has been very successful for fine-tuning, but it struggles with pre-training. Starting from scratch, W0W_0 is just a random initialization, and everything the model can learn is locked into rr dimensions; refinements like ReLoRA still need a stretch of full-rank warmup training first. The authors see two reasons. The optimal weights aren't necessarily low-rank, and the reparameterization itself changes the training dynamics. To take an extreme example, if the optimal solution of a linear regression y=Wxy = Wx is high-rank, then once WW is restricted to low rank, no optimizer can reach it.

So the authors asked a different question. The weights needn't be low-rank, but what about the gradients?

Solving the problem

Gradients become low-rank

The authors' theoretical starting point is a class of "reversible networks," which includes layers like linear layers (without bias), ReLU and leaky ReLU, along with their compositions and sums, residual networks included. For such networks, the weight gradient over a batch can be written as

Gt=1N∑i=1N(Ai−BiWtCi)G_t = \frac{1}{N}\sum_{i=1}^{N} \left(A_i - B_i W_t C_i\right)

Here BiB_i and CiC_i are positive semidefinite matrices. Lemma 3.3 proves that if AiA_i, BiB_i and CiC_i stay fixed after some point in time, then under plain SGD updates, the stable rank of GtG_t (∥G∥F2/∥G∥22\|G\|_F^2 / \|G\|_2^2, which you can think of as the "effective rank") falls exponentially, until only the part determined by the smallest eigensubspace remains. Corollary 3.4 gives a more concrete upper bound. When the gradient is a sum of outer products from NN data points and the input vectors have rank N′<nN' < n, the final stable rank is at most n−N′n - N'; and a sum of N′N' outer products itself has rank at most N′N', from which the paper concludes that in the long run the stable rank is at most n/2n/2.

Intuitively, a linear layer's gradient is a sum of outer products of "each data point's backpropagated error × each data point's input," so it only has a limited number of directions to begin with; after training for a while, most of its mass concentrates in just a few of them.

This theory covers only reversible networks, fixed coefficient matrices and SGD updates. For Transformers, the authors use a different framework in the appendix to prove only that the gradient of the feed-forward layer's first projection becomes low-rank; self-attention and Adam both lie outside the scope of the proofs.

Running Adam in a small projected space

Since the gradient concentrates in a few directions, keep optimizer state only along those directions. GaLore (Gradient Low-Rank Projection) is defined as

G~t=Pt ρt ⁣(Pt⊤GtQt)Qt⊤\tilde G_t = P_t\, \rho_t\!\left(P_t^\top G_t Q_t\right) Q_t^\top

where ρt\rho_t is any stateful optimizer, such as Adam. In practice, only a one-sided projection is used. For an m×nm \times n weight with m≤nm \le n, an SVD of the current gradient is taken every TT steps, and the top rr left singular vectors become the projection matrix,

Gt=USV⊤,Pt=U[:,:r]G_t = U S V^\top, \qquad P_t = U[:, :r]

Then at every step:

Rt=Pt⊤GtMt=β1Mt−1+(1−β1)RtVt=β2Vt−1+(1−β2)Rt2Nt=M^t/(V^t+ϵ)Wt=Wt−1+η⋅α PtNt\begin{aligned} R_t &= P_t^\top G_t \\ M_t &= \beta_1 M_{t-1} + (1-\beta_1) R_t \\ V_t &= \beta_2 V_{t-1} + (1-\beta_2) R_t^2 \\ N_t &= \hat M_t / (\sqrt{\hat V_t} + \epsilon) \\ W_t &= W_{t-1} + \eta \cdot \alpha\, P_t N_t \end{aligned}

(The paper defines GtG_t as the negative gradient, so the update is added to the weights.) α\alpha is a scale factor that plays a role like LoRA's α/r\alpha/r, but doesn't change with rr.

Let's walk through one weight from a LLaMA 7B feed-forward layer, of shape 4096×110084096 \times 11008; the paper's 7B experiment uses r=1024r = 1024. The gradient GtG_t is 4096×110084096 \times 11008; the projection matrix PtP_t is 4096×10244096 \times 1024; the projected RtR_t is only 1024×110081024 \times 11008, and so are Adam's MM and VV. Adam would originally store 2×4096×11008≈90.182 \times 4096 \times 11008 \approx 90.18 million numbers; now MM and VV together take 22.54 million, plus 4.19 million for PtP_t, about 26.74 million in all, roughly 30% of the original. The update itself is still a full 4096×110084096 \times 11008 matrix, and the weights stay full-rank.

Overall, GaLore stores mn+mr+2nrmn + mr + 2nr numbers for an m×nm \times n weight, while LoRA stores mn+3mr+3nrmn + 3mr + 3nr (the original weights plus two adapters, plus the adapters' two moments).

There's one more difference from LoRA. With rr at full rank, PtPt⊤=IP_t P_t^\top = I, and with plain SGD, GaLore's trajectory is identical to ordinary training; for LoRA, even if BABA is full-rank, optimizing BB and AA together follows a different trajectory.

The authors also prove convergence under a fixed projection (Theorem 3.8). The condition is ρt≡1\rho_t \equiv 1, that is, stateless SGD, and it requires the smallest eigenvalue of the projected subproblem to exceed a threshold set by continuity constants. The theorem suggests aiming the projection at the largest eigensubspace, which is the rationale for taking the top rr directions of the gradient's SVD. The authors also state explicitly that convergence only reflects good progress within this subspace, with no guarantee of overall performance.

Switching subspaces

Moving only within one fixed subspace, the change in the weights would never exceed rr dimensions, which brings back LoRA's old problem. GaLore switches subspaces every TT steps:

Wt=W0+ΔWT1+ΔWT2+⋯W_t = W_0 + \Delta W_{T_1} + \Delta W_{T_2} + \cdots

Each segment ΔWTi\Delta W_{T_i} is low-rank, but added up they can be full-rank.

Figure 2 from the paper: the weights start at W0, first update along the projected gradient within the blue plane (the first subspace) to reach W0 + ΔW_T1, then switch to the orange plane (the second subspace) and keep updating to reach W0 + ΔW_T1 + ΔW_T2.
Original Figure 2, Zhao et al., from PDF page 5, CC BY 4.0.

The blue plane is the subspace for the first T1T_1 steps, and the weights travel some distance within it; at T1T_1, the SVD is recomputed from the current gradient, switching to the orange plane, and the weights carry on from where they were.

How should TT be chosen? Switch too often, and the SVD overhead grows, the fixed projection the theorem requires is broken, and the moments Adam has accumulated get distorted because the basis keeps changing; switch too rarely, and you may get stuck in a subspace that no longer matters. The authors found little difference for TT anywhere between 50 and 1,000, use 200 for all pre-training, and estimate the total SVD overhead at under 10%. Also, when Algorithm 2 switches subspaces, MM and VV are carried over as they are, neither reset nor transformed into the new basis.

Two more tricks on top

GaLore compresses only the optimizer state, so the authors stack two existing tricks on top. One is replacing Adam with 8-bit Adam; "8-bit GaLore" below refers to this combination. The other is per-layer weight updates. As soon as backpropagation finishes computing a layer's gradient, that layer is updated immediately, so the whole model's gradients never have to be held until the end; this is the approach of the LOMO line of work.

Discussion

Pre-training from 60M to 1B

The experiments follow ReLoRA's setup, a LLaMA architecture (RMSNorm, SwiGLU) trained on C4 without repeating data, with sequence length 256, 131K tokens per step, and BF16 throughout. Each method at each size picks the best of five learning rates; GaLore uses learning rate 0.01, α=0.25\alpha = 0.25 and T=200T = 200 at every size, for an effective learning rate of 0.0025, larger than the 0.001 or below that the full-rank baselines usually use.

Method60M130M350M1B
Full-rank34.0625.0818.8015.56
GaLore34.8825.3618.9515.64
LoRA34.9933.9225.5819.21
ReLoRA37.0429.3729.0818.33
W=BAW = BA78.1845.5137.41142.53

Compiled from Table 2 of the paper; C4 validation perplexity, lower is better. Ranks are 128, 256, 256 and 512, respectively (model dimensions 512, 768, 1024 and 2048).

GaLore tracks full-rank closely at all four sizes, only 0.08 behind at 1B; LoRA and ReLoRA fall further behind as models grow. Factoring the weights directly into BABA and training from scratch is the worst, with a perplexity of 142.53 at 1B. For memory, the estimated parameters plus optimizer state at 1B are 7.80 GB for full-rank, 4.38 GB for GaLore and 6.17 GB for LoRA.

Keep two things in mind when reading this table. The memory here is a BF16 size estimated from parameter counts, excluding activations and gradients. And ReLoRA is compared with its full-rank warmup removed, which the authors say is for fairness, though its original design requires that warmup; LoRA, too, was originally designed for fine-tuning.

7B and that 4090

The 7B experiment ran for 150,000 steps and 19.7 billion tokens on 8 nodes with 64 A100s in total. Because compute was limited, 8-bit GaLore (r=1024r = 1024) and 8-bit Adam were each run only once, without tuning.

Method40K steps80K steps120K steps150K steps
8-bit GaLore17.9415.3914.9514.65
8-bit Adam18.0915.4714.8314.61

Compiled from Table 3 of the paper; C4 validation perplexity.

GaLore is slightly better early on, 8-bit Adam pulls ahead later, and the final gap is 0.04.

That 4090 shows up in a separate experiment. The authors measured actual memory on a single device, with no activation checkpointing, no offloading, and a token batch size of 256.

Figure 1 from the paper: estimated memory for six settings when pre-training LLaMA 7B, with stacked bars for weights, activations, optimizer state, weight gradients and other, in that order. BF16 AdamW is about 58 GB, Adafactor about 45 GB, AdamW without retained gradients and 8-bit Adam both about 44 GB, 8-bit Adam without retained gradients about 31 GB, and 8-bit GaLore without retained gradients about 22 GB, the only setting to the left of the RTX 4090's 24 GB dashed line.
Original Figure 1, Zhao et al., from PDF page 1, CC BY 4.0.

Look at the bottom two bars. 8-bit Adam without retained gradients is about 31 GB; switching to 8-bit GaLore shortens the optimizer-state segment (the darker green) considerably, for about 22 GB in total, to the left of the red dashed line at 24 GB. Compared with 8-bit Adam, the low-rank projection saves 9.6 GB of optimizer state, and per-layer updates save 13.5 GB of weight gradients; compared with BF16 AdamW, the total saving is 63.3%.

So "pre-training 7B on a 4090" measures whether the memory fits. The measurement used a batch of only 256 tokens (at most about 500 without checkpointing), and even with activation checkpointing it only goes up to 4,096; the 7B experiment the paper actually ran to completion used 64 A100s. The authors envision data parallelism across many consumer GPUs, which the paper never actually ran.

Speed and hyperparameters

Saving memory costs time. In measurements on LLaMA 1B with a token batch of 256, 8-bit GaLore with per-layer updates processes 1,019.63 tokens per second against 1,205.31 for 8-bit Adam, which the authors call about 17% overhead; with per-layer updates turned off, GaLore goes back up to 1,109.38.

Figure 5 from the paper: ablations on LLaMA 130M, with perplexity on the vertical axis. In the left plot, the horizontal axis is the subspace update frequency (about 5 to 20k steps), and the four curves for rank 64, 128, 256 and 512 are all U-shaped, bottoming out around 250 to 500; in the right plot, the horizontal axis is rank (16 to 512), and the five lines are 10k to 80k training steps, with higher rank giving lower perplexity and more steps giving lower curves.
Original Figure 5, Zhao et al., from PDF page 9, CC BY 4.0.

The left plot is the update frequency TT. All four ranks are U-shaped, getting worse if you switch too often or too rarely, with the bottom around one switch every 250 to 500 steps; the smaller the rank, the more often it needs to switch. The right plot is rank. Within a certain range, lowering the rank only slows convergence somewhat; rank 128 at 80,000 steps beats rank 512 at 20,000. In other words, when memory is short, you can trade time for it.

Fine-tuning

The fine-tuning comparisons are much milder than in pre-training. For RoBERTa-Base on GLUE at rank 4, GaLore averages 85.89, LoRA 85.61 and full fine-tuning 86.28; at rank 8 it's 85.94 versus 85.93, essentially a tie. The memory in the table, 253M versus 257M, again counts parameters plus optimizer state. In the appendix, for LLaMA-7B fine-tuned on OpenAssistant conversation data, test perplexity is 2.95 for GaLore and 2.94 for LoRA; the clearer gaps are BERT-Base on SQuAD (F1 88.29 versus 86.11) and Phi-2 fine-tuned on OpenAssistant (perplexity 3.83 versus 4.24).

Also, GaLore updates the full weights at every step, so what you get after training is an entire new model; LoRA lets you store just a small adapter that you can attach or detach at any time. Which is more convenient depends on your deployment needs.

Later scrutiny

The cost of SVD. The paper estimates SVD overhead at under 10%, but people who measured it see things differently. The author of LLaMA-Factory ran a third-party benchmark the day after GaLore's release. Fine-tuning LLaMA-2 7B on an A100 with activation checkpointing on, 16-bit GaLore without retained gradients needed only 18 GB against 40 GB for 8-bit AdamW, and throughput on ordinary steps held up too; but a note under the table says that each SVD on 7B takes about 10 minutes, which wasn't counted. Later, the team behind APOLLO (including two of GaLore's co-authors) also reported that a single subspace update on LLaMA-7B takes about 10 minutes, with throughput dropping every 200 steps. In the Hugging Face integration, layer-wise mode runs only on a single GPU, doesn't support DDP, and gradient clipping and DeepSpeed may not work properly.

Convergence. A paper by He et al. in October 2024 gave an explicit counterexample, proving that under standard stochastic optimization assumptions, GaLore doesn't necessarily converge to a stationary point. The reason is that SVD picks only the largest directions; late in training the true gradient gets small while the noise remains, and the top rr singular vectors may capture nothing but noise. GaLore has guarantees only when there's no noise, when the batch grows with the step count (for example Θ(T)\Theta(\sqrt T)), or when the noise is the same in every direction. Their proposed GoLore uses random projections instead, and they recommend using GaLore early on and switching to GoLore for the final stretch.

Scale. In April 2025, two of GaLore's authors, Jiawei Zhao and Yuandong Tian, published GaLore 2 together with Meta's PyTorch team. They state that each SVD for LLaMA 7B takes up to 20 minutes, and switching to randomized SVD made it 15 times faster; they also note that SVD signs aren't unique, so the projections from two decompositions may not line up. This time they trained LLaMA 7B from scratch on 500 billion tokens across 32 nodes with 256 H100s, with 8-bit Adam as the baseline. GaLore trailed slightly for the first 150 billion tokens, pulled ahead around 200 billion, and the two ended up comparable.

Conclusion

GaLore's most fundamental difference from LoRA is that it keeps the weights full-rank, puts only the optimizer state into the gradient's low-rank subspace, and switches subspaces periodically. In pre-training up to 1B, it trails full-rank Adam by only a few tenths of perplexity, far better than LoRA; 7B was run only once, and ended up about the same as 8-bit Adam. The price is an SVD every few hundred steps, and a theory that holds only under specific conditions.

If you want full-parameter fine-tuning on a single consumer GPU, GaLore is worth a try. Before you start, measure how long one SVD takes, and check whether your existing multi-GPU setup still works once per-layer updates are on. If you just want to save memory for fine-tuning, LoRA or QLoRA is usually less hassle.

And that 4090? GaLore 2's scale validation used 256 H100s.