Back to paper library
arXiv 2021JMLR 2022Language & TransformersPaper notes

Switch Transformer

Let the Experts Speak

Original paper

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Paper authors

William Fedus · Barret Zoph · Noam Shazeer

First submitted to arXiv
arXiv:2101.03961
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{transformersswitchtransformer2021,
  title = {{Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity}},
  author = {William Fedus and Barret Zoph and Noam Shazeer},
  year = {2021},
  eprint = {2101.03961},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.03961}
}
On this page

The effectiveness of large language models (LLMs) is well known.

However, the approach of continually increasing dense parameters will eventually hit a limit.

Defining the Problem

One idea, as we saw in the LoRA note, is to make fine-tuning cheaper. That does not solve the cost of training a large dense model from the start. Switch Transformer asks whether total parameters can grow without a proportional increase in computation per token.

The fundamental issue lies in the fact that the model architecture we use is computationally dense, leading to high computational costs.

Dense attention has quadratic attention cost O(n2)O(n^2) in sequence length nn, not exponential growth in model parameters. Switch sparsifies FFNs; it does not remove dense attention’s quadratic cost.

Another solution direction is the Mixtures of Experts (MoE) model.

MoE dates to the 1990s. The 2017 Google work scaled sparsely gated MoE to large neural networks and is an important predecessor here.

MoE decomposes the model into multiple "experts," with a "router" assigning each token to the most suitable expert based on the token's characteristics. This "sparse activation" strategy allows for a significant increase in the number of model parameters without adding extra computation per data point.

In other words, the goal of MoE is to build a model that is "massive in size but only activates a small portion of its components for each computation," aiming to achieve better performance at the same hardware cost.

The authors argue that the MoE model uses a "top-kk" expert routing strategy (k>1k > 1), which, while providing the router with multiple experts to choose from, also introduces relatively complex computation and communication costs. This process can likely be simplified:

  • As the saying goes, too many cooks spoil the broth, so here, just one "expert" is enough!

Solving the Problem

Model Architecture

From the model architecture diagram, we can clearly see the design concept of the Switch Transformer:

  1. Replacing Traditional FFN with Sparse Expert Layers (Switch FFN):

    In a typical Transformer, each token goes through the same set of FFN weights for feedforward computation. However, in the Switch Transformer, the FFN is decomposed into multiple "experts," each of which is an independent FFN.

  2. The Role of the Router: The diagram shows a router that dynamically selects which expert will handle each token based on its characteristics (e.g., the input word vector representation).

    • For example, in the diagram, there are two tokens (x1 = "More" and x2 = "Parameters"). The router will process and evaluate each token separately, then assign x1 to a specific expert and x2 to another (or the same) expert for processing.
  3. Sparsity Feature: Sparsity means activating one of many FFNs per token. Dense FFNs also operate separately on each token. Adding experts keeps per-token FFN FLOPs roughly constant, but routing, communication and storage of all expert weights still cost resources.

  4. Output and Router Weights (Gate Value): The selected expert output is multiplied by its router softmax probability pi(x)p_i(x). This retains a learnable continuous gate; it is not calibrated prediction confidence, nor is the sole selected gate renormalized to 1.

This is the dynamic routing mechanism of the Switch Transformer, where multiple experts each handle individual tokens.

Expert Capacity

When we have multiple experts processing a batch of tokens, we set a maximum token capacity limit for each expert.

This limit is determined by the "expert capacity," which is calculated as:

expert capacity=total tokens in batchnum experts×capacity factor\text{expert capacity} = \frac{\text{total tokens in batch}}{\text{num experts}} \times \text{capacity factor}

In other words, the "slot" number for each expert is fixed, and the "capacity factor" adjusts the additional space.

Each token is processed by a router, which chooses an expert based on a probability distribution. Ideally, assignments are balanced and no expert exceeds its capacity; with a capacity factor above 1, some slots can still remain empty.

When an expert overflows, excess tokens skip that expert computation and pass through the residual connection, shown by the red dashed lines. They are not deleted from the sequence.

Increasing the capacity factor allows for more space, reducing the likelihood of overflow. However, this also means that in some cases, parts of the experts' space may remain empty (as shown by the white spaces in the diagram), which leads to more unnecessary computation and communication costs.

Therefore, Switch Transformer needs to balance the "expert capacity" and "token routing" to achieve optimal performance.

Load Balancing Loss

To ensure the router does not repeatedly assign tokens to the same expert, Switch Transformer introduces a differentiable load balancing loss.

The basic idea is to make the number of tokens assigned to each expert and the total probability assigned by the router to each expert as even as possible. Let fif_i be the fraction of tokens assigned to expert ii and PiP_i be the average routing probability for that expert, i.e.,

fi=1T∑x∈B1{arg⁡max⁡p(x)=i},f_i = \frac{1}{T} \sum_{x \in B} 1\{\arg\max p(x) = i\}, Pi=1T∑x∈Bpi(x).P_i = \frac{1}{T} \sum_{x \in B} p_i(x).

If we want all experts to be evenly distributed, the ideal situation is fi≈1Nf_i \approx \frac{1}{N} and Pi≈1NP_i \approx \frac{1}{N}. Therefore, the auxiliary loss function is defined as:

lossaux=α⋅N∑i=1NfiPi,\text{loss}_{\text{aux}} = \alpha \cdot N \sum_{i=1}^{N} f_i P_i,

Here α=10−2\alpha=10^{-2}. When both distributions are uniform, the loss is α\alpha; the factor NN keeps this reference value independent of expert count. The argmax assignments in fif_i are not differentiable; gradients flow through PiP_i. This is a balancing surrogate, not a general rule that the inner product of any two distributions decreases with uniformity.

This mechanism provides additional gradients during training, encouraging the model to achieve more even load distribution.

Improved Training and Fine-Tuning

The sparse expert model in Switch Transformer may cause instability during training, especially with hard-switching routing decisions and the use of low-precision formats (like bfloat16).

The authors propose two improvements:

  1. Selective Precision Management: Use float32 precision locally within the routing function, only performing computations on the device in float32, then converting the results back to bfloat16. This approach retains the speed of bfloat16 training while maintaining the stability of float32.
  2. Smaller Parameter Initialization: The truncated normal has standard deviation σ=s/n\sigma=\sqrt{s/n}, where nn is fan-in. Reducing ss from 1 to 0.1 reduces the standard deviation by 1/101/\sqrt{10}, not by a factor of ten.

In addition, to address the overfitting problem during the fine-tuning stage, the authors propose a simple but effective method to alleviate this issue:

  • Increase the dropout within the expert layers, known as "Expert dropout."

The tested setting uses dropout 0.1 outside experts and 0.4 within expert FFNs, rather than a separate rate for every expert. On four smaller downstream tasks, this improves fine-tuning; Table 4 shows that increasing dropout throughout the model does not provide the same benefit.

Discussion

Comparison with Dense Models

This figure compares multilingual Switch-Base with mT5-Base on mC4 pretraining. The pretraining metric improves for all 101 languages.

The multilingual evidence here concerns that pretraining comparison. A product task in a particular language still needs its own evaluation.

Impact of the Number of Experts

The authors discuss several scaling characteristics, starting with the impact of the number of experts on model performance.

  • Left Diagram:

    • The horizontal axis represents the model's number of parameters (mainly increased by adding more experts).
    • The vertical axis shows test loss; lower is better.

    The point in the top left corresponds to T5-Base (approximately 223 million parameters). As the number of experts increases (from 2 to 256 experts), the total parameter count increases significantly, but the computation remains approximately the same.

    In this fixed-compute comparison, increasing the number of experts reduces test loss.


  • Right Diagram:

    • The horizontal axis is the number of training steps.
    • The vertical axis represents the negative log perplexity, where higher values indicate lower perplexity (better model performance).

    The diagram compares the dense T5-Base baseline (purple) with Switch-Base using 16, 32, 64 and 128 experts. At the same number of steps and similar FLOPs per example, the Switch models achieve higher negative log perplexity, showing better sample efficiency in this experiment.

In these fixed-step and fixed-compute comparisons, adding experts (and thus parameters) goes along with lower test loss and better sample efficiency. That is the scaling result the authors want us to notice.

Fixed Computation and Training Time

This comparison uses 32 TPUv3 cores and matched FLOPs per example to measure wall-clock time to the same pretraining quality.

The results, as shown in the above figure, indicate that, compared to traditional dense Transformers, the Switch Transformer (especially with 64 experts) achieves the same quality performance as T5-Base in much less time, even requiring only one-seventh of the time needed by T5-Base to reach the same perplexity level. Moreover, after reaching that quality threshold, the Switch Transformer continues to improve its model performance.

Model and Data Parallelism

The authors further explore how to balance resources in large-scale training scenarios, considering "data parallelism," "model parallelism," and "expert parallelism."

  1. Data Parallelism: Each core holds the same model replica and processes a different minibatch. Gradients are synchronized for each parameter update, not once after the entire training run; this communication can overlap backpropagation. A full replica must still fit in device memory.

  2. Model Parallelism: The model parameters (such as large weight matrices in FFN layers) are split across different cores, with each core storing and computing only a portion of the weights. This allows for training larger models since the full weight matrix no longer needs to fit into a single core. The trade-off is that cores need to frequently communicate during the computation. For example, FFN inputs and outputs need to be passed and aggregated between cores, as each core computes only a part of the output.

  3. Expert Parallelism: This is one of the key features of the Switch Transformer. The model is split into multiple "experts," each running on a different core, and each token is processed by only one expert based on the routing mechanism. This allows for increasing the number of parameters without significantly increasing the computation load. However, as the number of experts increases beyond a certain point, the benefits diminish, so data and model parallelism need to be combined to achieve optimal resource usage.

The detailed schematic is shown below:

This diagram uses a 4×4 dashed grid to illustrate how model weights and data are distributed across different computation cores in a distributed training setup. The diagram is divided into two parts:

  • Upper Part: The shapes show how weights are placed across cores; the same color denotes the same weights. Data parallelism replicates those weights, model parallelism splits one matrix across cores, and expert parallelism places different experts on different cores. The diagram holds the parameter count per core fixed, so a larger matrix also increases computation per token.

  • Lower Part: The colors show which cores hold the same or different data. Each core receives the same token count, but model-parallel cores work on the same data shard while data-parallel cores process different shards. Expert parallelism also sends each token across cores to its chosen expert.

The large models are Switch-XXL (395B parameters) and Switch-C (about 1.6T). Only Switch-XXL matches T5-XXL’s 6.3T FLOPs per sequence; Switch-C uses 890B. Both improve C4 pretraining perplexity in Table 9, but these gains do not fully carry over to downstream reasoning tasks.

These experiments show that this parameter scale was trainable in the authors’ setup and improved C4 pretraining sample efficiency. Switch-XXL was occasionally unstable, while the authors did not observe the same issue for Switch-C; stability therefore depends on the specific architecture and compute configuration.

Is the Success Due to the Large Number of Parameters?

Yes, the authors admit that the advantage of the Switch Transformer comes from its massive parameter count:

  • But this is a deliberate design strategy!

Increasing the number of parameters while maintaining the same computational budget leads to better sample efficiency and performance. In contrast to the traditional approach of scaling by increasing FLOPs (computational load), this method shows its value by achieving "better results with the same time and computational resources."

Can Small Models Benefit?

The authors emphasize that while Switch Transformers focus on extremely large models (hundreds of billions or even trillions of parameters), they can still run on standard GPUs/TPUs with a small number of experts (e.g., 2 experts) and still provide performance gains. In other words, researchers and users with small or medium-scale models can also benefit from this method.

Why Aren't Sparse Models Widely Adopted?

In the paper’s historical context, dense models had mature hardware support, while sparse models faced routing, inter-device communication and training-stability barriers.

Switch Transformer has made significant improvements in these areas, lowering the barriers for users to adopt sparse models.

Conclusion

Switch Transformer simplifies MoE with top-1 routing and demonstrates scaling total parameters at roughly fixed per-token FLOPs. Capacity is not free: storing weights, routing communication and large-scale training stability remain practical costs.

Compared to dense model baselines, Switch Transformers show outstanding performance in multiple natural language processing tasks and various training modes, achieving significant speed-ups.

The authors believe that the application of "sparse models" is definitely a direction worth exploring further and hope this research will bring more attention to this field.