Back to paper library
arXiv 2024DeepSeekPaper notes

DeepSeek-V2

Cost-Effective Optimization

Original paper

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

Paper authors

DeepSeek-AI

Publication
Publication venue not yet verified
First submitted to arXiv
arXiv:2405.04434
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{deepseekdeepseekv22024,
  title = {{DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model}},
  author = {DeepSeek-AI},
  year = {2024},
  eprint = {2405.04434},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.04434v5}
}
On this page

The previous paper was discussing testing the effects of MoE, and not even three months later, here comes another paper.

This paper has 52 pages, with a large amount of experimental data and technical details. As usual, we'll focus on the key points.

Defining the Problem

In recent years, with the rapid development of large language models, the increase in the number of parameters has indeed brought about significant improvements in model performance, even showcasing some "emergent capabilities," giving us hope of moving towards general artificial intelligence.

However, this improvement often comes with a huge consumption of computational resources, which not only makes training extremely expensive but also results in efficiency loss during inference due to computational and memory bottlenecks. This, in turn, affects the widespread adoption and deployment of these models in real-world applications.

Therefore, the DeepSeek research team focused on the computational bottleneck and efficiency issues and proposed the DeepSeek-V2 model, aiming to improve computational efficiency and reduce training and inference costs while maintaining strong performance.

Solving the Problem

DeepSeek-V2 still adopts the standard Transformer structure, where each Transformer block contains an attention module and a feed-forward network (FFN).

  • Attention Module: The research team proposed MLA, which uses low-rank key-value joint compression to address the KV cache bottleneck faced by traditional MHA during generation.
  • Feed-Forward Network: The DeepSeekMoE architecture is used, which is a high-performance Mixture-of-Experts model that allows training powerful models within limited costs.

Other details (such as Layer Normalization and activation functions) follow the setup of DeepSeek 67B. The overall architecture is shown in the figure below:

Paper Figure 2: the MLA and DeepSeekMoE architecture.

Multi-head Latent Attention

Cropped from paper Figure 2: the hatched vectors cached during MLA inference include the latent KV vector and decoupled positional key.

In the standard multi-head attention mechanism (MHA), assume:

  • dd: embedding dimension
  • nhn_h: number of attention heads
  • dhd_h: dimension of each head
  • ht∈Rdh_t \in \mathbb{R}^{d}: input representation of the tt-th token

Using three projection matrices WQW_Q, WKW_K, and WVW_V, each of size Rdhnh×d\mathbb{R}^{d_h n_h \times d}, the query (q), key (k), and value (v) are computed:

qt=WQhtkt=WKhtvt=WVht\begin{aligned} q_t &= W_Q h_t\\[1mm] k_t &= W_K h_t\\[1mm] v_t &= W_V h_t \end{aligned}

These vectors are then split into nhn_h heads as follows:

[qt,1; qt,2; … ; qt,nh]=qt[q_{t,1};\, q_{t,2};\, \dots; \, q_{t,n_h}] = q_t [kt,1; kt,2; … ; kt,nh]=kt[k_{t,1};\, k_{t,2};\, \dots; \, k_{t,n_h}] = k_t [vt,1; vt,2; … ; vt,nh]=vt[v_{t,1};\, v_{t,2};\, \dots; \, v_{t,n_h}] = v_t

Each head then performs the attention calculation:

ot,i=∑j=1tSoftmaxj ⁣(qt,i⊤kj,idh)vj,io_{t,i} = \sum_{j=1}^{t} \text{Softmax}_j\!\left(\frac{q_{t,i}^\top k_{j,i}}{\sqrt{d_h}}\right) v_{j,i}

Finally, the outputs of the heads are combined through an output projection matrix WO∈Rd×dhnhW_O \in \mathbb{R}^{d \times d_h n_h}:

ut=WO[ot,1; ot,2; … ; ot,nh]u_t = W_O [o_{t,1};\, o_{t,2};\, \dots; \, o_{t,n_h}]

Up to this point, this is the standard MHA computation process.

However, during inference, caching the keys and values requires 2nhdh2 n_h d_h elements per token per layer. With fixed model dimensions and layers, KV-cache memory grows linearly with the cached sequence length, making it a significant bottleneck.

Thus, the core of MLA is to reduce the KV cache using "low-rank compression."

The specific steps are as follows:

  1. Key-Value Compression: For each token, we first compute its compressed latent representation:

    cKV,t=WDKVhtc_{KV,t} = W_{DKV} h_t

    where cKV,t∈Rdcc_{KV,t} \in \mathbb{R}^{d_c}, dc≪dhnhd_c \ll d_h n_h, and WDKV∈Rdc×dW_{DKV} \in \mathbb{R}^{d_c \times d} is the down-projection matrix.

  2. Up-Projection to Recover Keys and Values: The keys and values are recovered from the compressed representation:

    kC,t=WUKcKV,tk_C,t = W_{UK} c_{KV,t} vC,t=WUVcKV,tv_C,t = W_{UV} c_{KV,t}

    where WUKW_{UK} and WUVW_{UV} are the up-projection matrices for the key and value, respectively, with dimensions Rdhnh×dc\mathbb{R}^{d_h n_h \times d_c}.

Before adding positional encoding, inference can cache only cKV,tc_{KV,t}, or dcd_c elements per token per layer. With decoupled RoPE, the shared positional key ktRk^R_t must also be cached: the full MLA budget is dc+dhR=512+64=576d_c+d_h^R=512+64=576 elements per token per layer in this configuration.

Additionally, because WUKW_{UK} can be absorbed into WQW_Q, and WUVW_{UV} can be absorbed into WOW_O, there is no need for explicit key and value computation during the actual attention calculation.

Finally, to reduce activation memory during training, the research team also applied a similar low-rank compression to the query vectors:

cQ,t=WDQhtc_{Q,t} = W_{DQ} h_t qC,t=WUQcQ,tq_C,t = W_{UQ} c_{Q,t}

where cQ,t∈Rdc′c_{Q,t} \in \mathbb{R}^{d'_c} is the compressed representation of the query, and dc′≪dhnhd'_c \ll d_h n_h.

Decoupling Rotary Position Embedding

Rotary Position Embedding (RoPE) is a method for incorporating position information in Transformer models, but it is sensitive to both keys and queries. In the case of low-rank compression, if RoPE is directly applied to the compressed key kC,tk_C,t, it will couple the up-projection matrix WUKW_{UK} with the positional information, thus breaking the earlier absorption optimization and forcing the need to recompute the keys for previous tokens during inference.

The research team proposed a decoupled RoPE strategy. Specifically, they added extra multi-head queries qR,t,i∈RdhRq_{R,t,i} \in \mathbb{R}^{d^R_h} and shared keys kR,t∈RdhRk_{R,t} \in \mathbb{R}^{d^R_h} to carry the positional information of RoPE.

Then, they computed the following:

[qR,t,1; qR,t,2; … ; qR,t,nh]=qR,t=RoPE(WQRcQ,t)[q_{R,t,1};\, q_{R,t,2};\, \dots; \, q_{R,t,n_h}] = q_{R,t} = \text{RoPE}(W_{QR} c_{Q,t}) kR,t=RoPE(WKRht)k_{R,t} = \text{RoPE}(W_{KR} h_t)

Where WQR∈RdhRnh×dc′W_{QR} \in \mathbb{R}^{d^R_h n_h \times d'_c} and WKR∈RdhR×dW_{KR} \in \mathbb{R}^{d^R_h \times d} are the matrices used to generate decoupled queries and keys.

Finally, the compressed part and the decoupled part are concatenated:

qt,i=[qC,t,i; qR,t,i]q_{t,i} = [q_{C,t,i};\, q_{R,t,i}] kt,i=[kC,t,i; kR,t]k_{t,i} = [k_{C,t,i};\, k_{R,t}]

And the attention calculation is performed:

ot,i=∑j=1tSoftmaxj ⁣(qt,i⊤kj,idh+dhR)vC,j,io_{t,i} = \sum_{j=1}^{t} \text{Softmax}_j\!\left(\frac{q_{t,i}^\top k_{j,i}}{\sqrt{d_h + d^R_h}}\right) v_{C,j,i}

Finally, the outputs of the heads are combined using WOW_O:

ut=WO[ot,1; ot,2; … ; ot,nh]u_t = W_O [o_{t,1};\, o_{t,2};\, \dots; \, o_{t,n_h}]

The independent positional key ktRk^R_t leaves the content-key up-projection absorbable into the query side. Inference caches ctKVc^{KV}_t and ktRk^R_t, rather than full multi-head K/V vectors for every prefix token.

DeepSeekMoE

Cropped from paper Figure 2: shared and routed experts in DeepSeekMoE.

Unlike traditional MoE models, DeepSeekMoE performs a more detailed segmentation of the experts: splitting the FFN module into two parts:

  • Shared Experts: A fixed number of experts that each token will use.
  • Routed Experts: Experts that are dynamically selected to participate in the computation.

For the tt-th token, its FFN input is utu_t, and the final FFN output ht′h'_t is defined as:

ht′=ut+∑i=1NsFFNi(s)(ut)+∑i=1Nrgi,t FFNi(r)(ut)h'_t = u_t + \sum_{i=1}^{N_s} \text{FFN}^{(s)}_i(u_t) + \sum_{i=1}^{N_r} g_{i,t} \,\text{FFN}^{(r)}_i(u_t)

Where:

  • NsN_s is the number of shared experts,
  • NrN_r is the number of routed experts,
  • FFNi(s)(⋅)\text{FFN}^{(s)}_i(\cdot) and FFNi(r)(⋅)\text{FFN}^{(r)}_i(\cdot) represent the feed-forward operations of the ii-th shared expert and routed expert, respectively.

The selection of routed experts depends on a gating mechanism, defined as follows:

gi,t={si,t,if si,t∈Topk({sj,t∣1≤j≤Nr},Kr),0,otherwise,g_{i,t} = \begin{cases} s_{i,t}, & \text{if } s_{i,t} \in \text{Topk}(\{ s_{j,t} \mid 1 \le j \le N_r\}, K_r),\\[1mm] 0, & \text{otherwise}, \end{cases}

Where KrK_r is the number of active routed experts, and si,ts_{i,t} is the affinity score between the token and the ii-th routed expert, computed as:

si,t=Softmaxi(ut⊤ei)s_{i,t} = \text{Softmax}_i\bigl(u_t^\top e_i\bigr)

Here eie_i is a learned routing vector for expert ii, called a centroid in the paper; it is not a fixed centroid computed by clustering the dataset.

In MoE models, when experts are distributed across multiple devices, the routing decision for each token may involve cross-device communication. If a large number of experts are activated, this will incur additional communication costs. To control this, DeepSeekMoE uses a device-limited routing mechanism:

  1. For each token, select MM devices with the highest affinity scores across all devices.
  2. Then, perform Top-K expert selection only within these MM devices.

Experimental observations show that when M≥3M \ge 3, this limitation significantly reduces cross-device communication while maintaining performance close to the unrestricted Top-K selection.

The core of this strategy is to balance routing quality with communication costs between devices, achieving high computational efficiency in large-scale distributed environments.

Auxiliary Balance Loss

Since MoE models often experience situations where some experts are overactivated (or even completely unused), this can lead to "routing collapse" and reduce computational efficiency. To address this, DeepSeekMoE has designed three auxiliary losses to promote load balancing:

  1. Expert-Level Balance Loss

    LExpBal=α1∑i=1Nrfi PiL_{ExpBal} = \alpha_1 \sum_{i=1}^{N_r} f_i\, P_i

    Where:

    fi=NrKr T∑t=1T1(Token t selects Expert i)f_i = \frac{N_r}{K_r\,T} \sum_{t=1}^{T} 1(\text{Token } t \text{ selects Expert } i) Pi=1T∑t=1Tsi,tP_i = \frac{1}{T} \sum_{t=1}^{T} s_{i,t}
    • TT is the total number of tokens in the sequence,
    • 1(⋅)1(\cdot) is an indicator function,
    • α1\alpha_1 is a hyperparameter used to adjust the loss weight.

    This loss aims to balance the activation frequency and average affinity score of each routed expert.


  1. Device-Level Balance Loss

    The routed experts are grouped into DD groups (each group EiE_i placed on the same device). The loss is defined as:

    LDevBal=α2∑i=1Dfi′ Pi′L_{DevBal} = \alpha_2 \sum_{i=1}^{D} f'_i\, P'_i

    Where:

    fi′=1∣Ei∣∑j∈Eifjf'_i = \frac{1}{|E_i|} \sum_{j \in E_i} f_j Pi′=∑j∈EiPjP'_i = \sum_{j \in E_i} P_j
    • α2\alpha_2 is the device-level balance hyperparameter.

    This helps ensure the computation load is balanced between devices, preventing any one device from becoming overloaded and causing bottlenecks.


  1. Communication Balance Loss

    Even with device-limited routing, if some devices receive significantly more tokens than others, this can affect communication efficiency. Therefore, the following loss is introduced:

    LCommBal=α3∑i=1Dfi′′ Pi′′L_{CommBal} = \alpha_3 \sum_{i=1}^{D} f''_i\, P''_i

    Where:

    fi′′=DM T∑t=1T1(Token t is sent to device i)f''_i = \frac{D}{M\,T} \sum_{t=1}^{T} 1(\text{Token } t \text{ is sent to device } i) Pi′′=∑j∈EiPjP''_i = \sum_{j \in E_i} P_j
    • α3\alpha_3 is the communication balance hyperparameter.

    This design encourages each device to maintain a similar level of communication, improving computational coordination efficiency across devices.


Finally, in addition to balance losses, the research team introduced a device-level token dropping strategy to further mitigate the computational waste caused by load imbalances. The method is as follows:

  1. Calculate the average computation budget for each device so that the capacity factor for each device is set to 1.0.
  2. Drop the tokens with the lowest affinity scores on each device until the computation budget is reached.
  3. Protect all tokens belonging to about 10% of training sequences; the paper selects sequences, not a random 10% sample of individual tokens.

The paper uses token dropping for training speed, while protecting every token in about 10% of training sequences to retain a no-drop path. Its evaluation does not drop tokens.

Training Strategy

The paper maintains the same data processing pipeline as DeepSeek 67B, but with the following enhancements:

  • Expanded Data Volume: The cleaning process was optimized to recover a large amount of previously deleted data.
  • Improved Data Quality: More high-quality data was added, especially Chinese data; in the resulting 8.1T-token corpus, the number of Chinese tokens is about 12% higher than the number of English tokens, rather than a claim about tokenization length per sentence. Additionally, quality-based filtering algorithms were improved to remove a large amount of useless information while retaining valuable content.
  • Filtered Controversial Content: This strategy aims to reduce biases from specific regional cultures.

The tokenizer uses the same Byte-level Byte-Pair Encoding (BBPE) algorithm as DeepSeek 67B, with a vocabulary size set to 100K, resulting in a total pre-training corpus of 8.1T tokens.

Model hyperparameters are as follows:

  • Number of Transformer layers: 60 layers
  • Hidden layer dimension: 5120
  • Standard deviation of randomly initialized parameters: 0.006
  • MLA-related:
    • Number of attention heads nhn_h: 128
    • Dimension of each head dhd_h: 128
    • KV compression dimension dcd_c: 512
    • Query compression dimension dc′d'_c: 1536
    • Decoupled query and key head dimension dhRd^R_h: 64

MoE layer configuration is as follows: except for the first layer, all other FFN layers are replaced with MoE layers.

  • Each MoE layer contains 2 shared experts and 160 routed experts
  • The hidden dimension of each expert is 1536
  • Each token activates 6 routed experts
  • To address the impact of low-rank compression and fine-grained expert segmentation on layer output size, additional RMS Norm layers and scaling factors were introduced to ensure training stability.

Finally, DeepSeek-V2 has a total of 236B parameters, but only 21B parameters are activated per token.

Training hyperparameters are as follows:

  • Optimizer: AdamW, hyperparameters β1=0.9\beta_1=0.9, β2=0.95\beta_2=0.95, weight decay = 0.1
  • Learning rate scheduling: Linear warmup for the first 2000 steps, then scaled by 0.316 at approximately 60% and 90% of the token count, with a maximum learning rate of 2.4×10⁻⁴
  • Gradient clipping set to 1.0
  • Batch size uses a progressive adjustment strategy, from 2304 to 9216, then remains at 9216
  • Maximum sequence length set to 4K, trained on 8.1T tokens
  • Other parallelization setups:
    • Pipeline parallelism is used to deploy different layers across different devices
    • Routed experts are evenly distributed across 8 devices (D=8D=8)
    • Device-limited routing is set to send each token to a maximum of 3 devices (M=3M=3)
    • Hyperparameters for the auxiliary balance losses are set to α1=0.003\alpha_1=0.003, α2=0.05\alpha_2=0.05, α3=0.02\alpha_3=0.02
    • Token-dropping strategy used during training for acceleration, but tokens are not dropped during evaluation

After the preliminary pre-training, the research team used the YaRN method to expand the model's default context window from 4K to 128K.

Particular attention was given to expanding the decoupled shared key ktRk^R_t responsible for carrying RoPE, with parameters set as: scale s=40s=40, α=1\alpha=1, β=32\beta=32, and a target maximum context length of 160K.

To accommodate the unique attention mechanism, the length scaling factor was adjusted, making the formula for t\sqrt{t} as follows:

t=0.0707ln⁡s+1\sqrt{t} = 0.0707 \ln s + 1

This aims to minimize perplexity.

During the long-context expansion phase, the model was trained for another 1000 steps at a sequence length of 32K and a batch size of 576 sequences. Although training used 32K sequences, it performed well in the paper’s "Needle In A Haystack" (NIAH) test at the evaluated context lengths up to 128K. That test does not cover every long-context task.

Model Alignment

To improve the alignment of the model with human preferences and further enhance the accuracy and satisfaction of responses, the authors performed reinforcement learning adjustments. The Group Relative Policy Optimization (GRPO) algorithm was used, which is particularly advantageous as it saves training costs by eliminating the need for a critic model of the same scale as the policy model. Instead, group scores are used to estimate the baseline.

The reinforcement learning training strategy is divided into two stages:

  • Stage 1: Reasoning Alignment

    In this stage, the authors trained a dedicated reward model RMreasoningR_{\text{Mreasoning}} for reasoning tasks, such as programming and mathematics:

    ri=RMreasoning(oi)r_i = R_{\text{Mreasoning}}(o_i)

    This helps the model continuously improve its problem-solving abilities for such tasks.

  • Stage 2: Human Preference Alignment

    In this stage, a multi-reward framework is used, with rewards obtained from three aspects:

    • A helpful reward model RMhelpfulR_{\text{Mhelpful}}
    • A safety reward model RMsafetyR_{\text{Msafety}}
    • A rule-based reward model RMruleR_{\text{Mrule}}

    Ultimately, for a given response oio_i, the reward value is calculated as:

    ri=c1⋅RMhelpful(oi)+c2⋅RMsafety(oi)+c3⋅RMrule(oi)r_i = c_1 \cdot R_{\text{Mhelpful}}(o_i) + c_2 \cdot R_{\text{Msafety}}(o_i) + c_3 \cdot R_{\text{Mrule}}(o_i)

    Where c1,c2,c3c_1, c_2, c_3 are the respective coefficients. This multi-reward design helps the model consider multiple quality metrics when generating responses.

To obtain reliable reward signals, the research team collected large amounts of preference data:

  • Preference data for programming tasks were based on compiler feedback,
  • Preference data for mathematics tasks were based on real labels.

The reward model was initialized using DeepSeek-V2 Chat (SFT) results and trained with point-to-point or pairwise loss.

Discussion

Due to space limitations, let's look at the results from AlignBench. If you are interested in other experimental results, you can refer to the original paper.

Paper Table 5: AlignBench Chinese leaderboard judged by GPT-4-0613.

The table ranks the models based on their overall performance scores on AlignBench, with higher-scoring models ranked higher. These are GPT-4-0613-judged Chinese open-ended scores, subject to that judge’s rubric and preferences; rankings should be read within the stated benchmark and model versions.

In this table, DeepSeek-V2 Chat (RL) scores 7.91 overall, above SFT at 7.74 and the other listed open models, including Qwen1.5 72B Chat at 7.19. The reasoning average rises from 7.30 to 7.45 and the language average from 8.17 to 8.36, but logical reasoning falls from 7.26 to 7.14.

For Chinese comprehension, RL has 8.28 versus GPT-4-1106-Preview’s 7.33, but below ERNIEBot-4.0-202404 at 8.53. Its 7.45 reasoning average is also below ERNIEBot’s 7.61 and GPT-4-1106-Preview’s 7.73.

This AlignBench table covers Chinese tasks. The separate English conversation results in paper Table 4 show RL above SFT on MT-Bench (8.97 versus 8.62) and AlpacaEval 2.0 length-controlled win rate (38.9 versus 30.0). The two tables show benefits and trade-offs for their respective tasks; English conversation performance cannot be inferred directly from AlignBench.

Conclusion

Paper Figure 1: capability, training cost, KV cache, and generation throughput comparisons.

DeepSeek-V2 is a large language model based on the MoE architecture, supporting up to 128K context length.

The model leverages two innovative techniques:

  • Multi-head Latent Attention: By using low-rank key-value joint compression, it greatly reduces the need for KV cache during inference, thus improving inference efficiency.
  • DeepSeekMoE: Through fine-grained expert segmentation and shared expert designs, it achieves the goal of training powerful models at lower costs.

Compared with DeepSeek LLM 67B under the paper’s training and deployment setup, DeepSeek-V2 reports:

  • A 42.5% reduction in training costs;
  • A 93.3% reduction in KV cache usage;
  • A 5.76x improvement in generation throughput.

Despite having a total of 236B parameters, each token only activates 21B parameters, which strikes a good balance between economy and performance, making it competitive among the open MoE comparisons at the paper’s publication date. The reported throughput depends on the paper’s deployment setup.