Back to paper library
arXiv 2024DeepSeekPaper notes

DeepSeek LLM

New Scaling Laws

Original paper

DeepSeek LLM: Scaling Open-Source Language Models with Longtermism

Paper authors

DeepSeek-AI

Publication
Publication venue not yet verified
First submitted to arXiv
arXiv:2401.02954
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{deepseekdeepseekllm2024,
  title = {{DeepSeek LLM: Scaling Open-Source Language Models with Longtermism}},
  author = {DeepSeek-AI},
  year = {2024},
  eprint = {2401.02954},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.02954v1}
}
On this page

Recently, DeepSeek has been in the spotlight, generating a lot of buzz, with even OpenAI stepping in to comment.

Given the immense impact they’ve made, let’s take some time to review their recent publications.

This paper is 48 pages long, filled with extensive experimental data and technical details, so we’ll focus on the key points.

Defining the Problem

The performance gap between open-source and closed-source models has long been a popular topic. Previous research has shown that open-source models generally perform significantly worse than closed-source models on most benchmarks, primarily due to the scale and resource limitations of open-source models.

To bridge this gap, the research team proposed the DeepSeek LLM series, a set of open-source language models ranging in size from 7B to 67B. To train these models, the research team challenged the existing Scaling Laws, arguing that past research findings may not be entirely accurate. Therefore, they introduced their own Scaling Law, which became the core of the paper, exploring the relationship between model size, data scale, and computational budget.

We will focus on the discussion of the Scaling Law and briefly introduce the design and experimental results of DeepSeek LLM.

Solving the Problem

Data Processing Strategy

The research team adopted a three-phase strategy: "Deduplication → Filtering → Remixing."

  • Deduplication:

    Paper Table 1: cross-dump Common Crawl deduplication ratios.

    The entire Common Crawl corpus was used for deduplication. As seen in the table above, deduplication across multiple dumps is more thorough than when applied to a single dump, providing the model with more unique learning samples.

  • Filtering: A rigorous document quality evaluation standard was designed, combining linguistic and semantic analysis to ensure data quality from both local and global perspectives.

  • Remixing: To address the issue of uneven data distribution, the team specifically enhanced data from underrepresented domains, ensuring that the dataset was more balanced and inclusive of diverse information.

In terms of Tokenizer design, the research team employed Byte-level Byte-Pair Encoding (BBPE), using pre-tokenization to avoid mixing different character categories (such as newlines, punctuation, and CJK symbols). Additionally, they split numbers into individual digits. The vocabulary was initially set to 100,000, with 15 special tokens and extra space up to 102,400, balancing computational efficiency in model training with future scalability needs.

Model Architecture Design

The model largely follows the LLaMA architecture, using a Pre-Norm structure with RMSNorm and SwiGLU as the activation function for the Feedforward Neural Network (FFN). Rotary Embedding is employed for position encoding to enhance sequence processing capability.

To reduce inference costs, the 67B model uses Grouped-Query Attention (GQA) to replace traditional Multi-Head Attention.

The smaller (7B) and larger (67B) models have 30 and 95 layers, respectively. The authors say this keeps their parameter counts close to those of other open-source models and makes pipeline partitioning easier. Rather than mainly widening the intermediate FFN layers, the team scaled the 67B model largely by adding depth.

Parameter Configuration

The model initialization standard deviation was set to 0.006, and the AdamW optimizer was used:

  • β1=0.9\beta_1=0.9
  • β2=0.95\beta_2=0.95
  • weight_decay=0.1

The learning rate scheduler employed a multi-step configuration, reaching its peak after 2000 warmup steps, then gradually decaying to 31.6% and 10% of the maximum value at 80% and 90% of the training tokens, respectively. This strategy allows for adjusting training scales under a fixed model size while reusing the previous stage’s training results, which is convenient for continued training. Overall, experiments show that, while the loss curves differ, the final performance is consistent with the cosine scheduler.

Gradient clipping was set to 1.0 to prevent gradient explosion and stabilize training.

Training was conducted using the HAI-LLM framework, which integrates data, tensors, sequences, and pipeline parallelism technologies, similar to Megatron’s design, effectively sharing the computational load of large-scale models.

Additionally, flash attention technology was introduced to improve hardware utilization, and ZeRO-1 technology was used to partition the optimizer state, further reducing memory consumption. Some layers (such as LayerNorm, GEMM, and Adam updates) were fused to accelerate training; mixed precision (bf16 training, fp32 gradient accumulation) and in-place cross-entropy computation were used to optimize memory usage.

Model weights and optimizer states are automatically saved every 5 minutes, ensuring that at most 5 minutes of training is lost in case of hardware or network failures. Training can also be resumed from different 3D parallel configurations to adapt to dynamic changes in computational resources.

For evaluation, the vLLM was used for generative tasks, while non-generative tasks were processed using continuous batch handling to minimize manual adjustments of batch size and token padding.

Scaling Laws

Hyperparameter Configuration

Paper Figure 2: training loss across batch sizes and learning rates at fixed compute budgets.

The research team first performed a grid search for a specific model scale (177M FLOPs/token) under a computational budget of C=1e17C = 1\mathrm{e}17, exploring different combinations of batch size BB and learning rate η\eta.

Based on the experimental results shown in the figure above, the research team observed that within a wide range of BB and η\eta, the loss remained close to its minimum. This suggests that within this larger parameter space, an "approximately optimal" set of hyperparameters could be found.

To further explore a larger computational budget range (from 1e171\mathrm{e}17 to 2e192\mathrm{e}19), the research team adopted a multi-step learning rate scheduler. This scheduler allows for the reuse of previously trained results after the first stage of training, enabling efficient training of multiple hyperparameter configurations. Since there is redundancy in the parameter space, the team defined that the hyperparameters are considered "approximately optimal" when the model's generalization error exceeds the minimum by no more than 0.25%.

Through extensive experiments and data fitting, the results showed:

  • Optimal Batch Size BoptB_{opt}: Increases gradually with the computational budget CC. This aligns with intuition, as more computational resources allow for larger batches.
  • Optimal Learning Rate ηopt\eta_{opt}: Decreases as CC increases. In other words, as the model size grows and the computational budget increases, a lower learning rate is needed to maintain stability.

Finally, the research team presented the following two formulas to describe this relationship:

ηopt=0.3118⋅C−0.1250\eta_{opt} = 0.3118 \cdot C^{-0.1250} Bopt=0.2920⋅C0.3271B_{opt} = 0.2920 \cdot C^{0.3271}

These formulas quantify the power-law relationships between optimal learning rate and optimal batch size with respect to the computational budget CC.

To check these formulas at a larger computational budget of C=1e20C = 1\mathrm{e}20, the research team used a model scale of 2.94B FLOPs/token. The results in the right-hand plot placed the fitted hyperparameters near the center of the observed near-optimal region for this setting.

Optimal Model and Data Scaling Estimates

After fitting the approximate optimal hyperparameter formulas, the research team began investigating how to optimally allocate model size and data scale:

That is, determining the best model size and data size to minimize error, given a specific computational budget CC.

The core goal was to identify two power-law relationships, where the optimal model size NoptN_{opt} is related to the computational budget as:

Nopt∝CaN_{opt} \propto C^{a}

And the optimal data size DoptD_{opt} (measured in the number of tokens in the data) follows:

Dopt∝CbD_{opt} \propto C^{b}

Here, aa and bb are the scaling exponents for the model and data, respectively.

Early works often used the number of model parameters to represent model size, and the relationship between computational budget and model/data size was approximately:

C=6N⋅DC = 6N \cdot D

Model size could be approximated by 6N16N_1 or 6N26N_2. However, these representations failed to fully account for the computational burden of attention operations, leading to errors of up to 50% in different model scales, especially for smaller models. These errors can significantly impact the statistical fitting of subsequent Scaling curves.

To overcome this issue, the research team introduced a new model size representation MM, which includes the computational cost of attention operations but excludes vocabulary calculations. The formula is:

M=72 nlayer dmodel2+12 nlayer dmodel lseqM = 72\, n_{layer}\, d_{model}^2 + 12\, n_{layer}\, d_{model}\, l_{seq}

Where:

  • nlayern_{layer}: Number of layers
  • dmodeld_{model}: Model width
  • lseql_{seq}: Sequence length

Using MM, the computational budget can be simplified as:

C=M⋅DC = M \cdot D

This makes it easier to estimate the optimal allocation of model and data across different configurations.

After introducing MM, the research team clarified their objective:

Given C=M⋅DC = M \cdot D, find the optimal model size MoptM_{opt} and optimal data size DoptD_{opt} such that validation loss is minimized (using MM, non-embedding FLOPs per token, as the model-scale variable).

This can be expressed as:

Mopt(C), Dopt(C)=arg⁡min⁡M,D s.t. C=M⋅DL(M,D)M_{opt}(C),\, D_{opt}(C) = \arg\min_{M,D \, \text{s.t.} \, C = M \cdot D} L(M,D)

To reduce experimental costs and fitting complexity, the research team used the IsoFLOP profile method introduced in Chinchilla:

  • Select 8 different computational budgets, ranging from 1e171\mathrm{e}17 to 3e203\mathrm{e}20.
  • For each budget, design about 10 different model and data size configurations.
  • For each configuration, hyperparameters are determined by the previously established formulas and generalization errors are calculated on an independent validation set (with 100M tokens).

Using this data, the research team plotted the IsoFLOP curve and the model/data scaling curves:

Paper Figure 4: IsoFLOP profiles and optimal model/data allocation, measured in validation bits-per-byte.

They finally obtained the optimal model and data scaling formulas:

Mopt=Mbase⋅CaM_{opt} = M_{base} \cdot C^{a}
  • Where Mbase=0.1715,  a=0.5243M_{base} = 0.1715,\; a = 0.5243.
Dopt=Dbase⋅CbD_{opt} = D_{base} \cdot C^{b}
  • Where Dbase=5.8316,  b=0.4757D_{base} = 5.8316,\; b = 0.4757.

Based on these fitting results, the research team successfully predicted the generalization errors for the DeepSeek LLM 7B and 67B models, as shown below:

Paper Figure 5: validation bits-per-byte for 7B/67B and the scaling-law prediction.

Experimental results show that small-scale experiments can accurately predict model performance under 1000 times the computational budget, providing strong support for large-scale model training.

Impact of Data Quality

The research team selected three datasets for comparative analysis:

  • Early Data: Early internal data.
  • Current Data: Current internal data.
  • OpenWebText2: The dataset used in previous scaling laws research.

Through comparisons, the evaluation showed that the quality of the current internal data is better than the early data, while OpenWebText2, due to its smaller size, can undergo more meticulous processing, and its data quality even surpasses that of the current internal data.

Using the Scaling Laws formulas from the previous section, the research team calculated the Scaling indices for different datasets, as shown in the table below:

Paper Table 4: model/data scaling exponents for different training-data distributions.

Specifically, OpenAI (using OpenWebText2) had coefficients of a=0.73a=0.73 and b=0.27b=0.27; Chinchilla (MassiveText) had coefficients of a=0.49a=0.49 and b=0.51b=0.51; and for their own data, early in-house data had a=0.450,b=0.550a=0.450, b=0.550; current in-house data had a=0.524,b=0.476a=0.524, b=0.476; and their OpenWebText2 run had a=0.578,b=0.422a=0.578, b=0.422.

From these results, it can be seen that as data quality improves, the model expansion index aa gradually increases, while the data expansion index bb decreases accordingly. Across these three data conditions, the authors’ quality ranking coincides with higher aa and lower bb: as compute grows, the fitted allocation shifts relatively toward model scale. Data quality alone is not isolated as the cause. This also explains why earlier studies showed significant differences in the optimal model and data allocation strategies across different datasets.

Model Alignment

The main goal of the model alignment (Alignment) process is to ensure that when the model generates responses, it provides useful information while avoiding harmful content.

Regarding the data, the research team collected approximately 1.5 million Chinese and English instruction data points, covering a wide range of topics. Of these, 1.2 million are considered helpful data, distributed as follows: general language tasks make up 31.2%, math problems 46.6%, and programming exercises 22.2%; an additional 300,000 data points focus on safety issues, covering various sensitive topics.

The alignment process is divided into two stages:

  • The first stage is supervised fine-tuning (Supervised Fine-Tuning, SFT).
  • The second stage uses direct preference optimization (Direct Preference Optimization, DPO).

In the supervised fine-tuning stage, different training epochs were used for different model sizes because larger models are more prone to overfitting. The experiment used benchmarks like GSM8K and HumanEval, finding that the performance of the 7B model continued to improve, while the 67B model reached its performance ceiling more quickly.

In the second stage, DPO was employed to further enhance the model's alignment. DPO has proven to be a simple and effective method, with its core being the adjustment of the model’s response generation direction using preference data.

In this stage, the research team constructed preference data for helpfulness and harmlessness. Specifically, they first collected multilingual prompts covering creative writing, Q&A, instruction following, and other areas, then used the DeepSeek Chat model to generate candidate responses for helpfulness preference data. Harmlessness preference data was constructed using a similar process.

Discussion

DeepSeek LLM has a lot of experimental results. We’ll mainly look at the open-ended Chinese responses in AlignBench and the multi-turn English conversations in MT-Bench; neither benchmark represents every real-world use case.

For other aspects, interested readers can refer to the original paper.

Chinese Open-Ended Evaluation

Paper Table 7: AlignBench leaderboard judged by GPT-4-0613.

The research team tested the model’s open-ended generative abilities on Chinese and English tasks.

For the Chinese evaluation, they used the AlignBench test set, which covers 8 major categories and 36 subcategories, with a total of 683 questions. Each question provides a prompt, reference answer, and scoring template, and GPT-4-0613 judges the responses. These are model-judge results.

During the evaluation process, the research team used the official AlignBench GitHub repository to implement the model evaluation and strictly adjusted the generation temperature parameters according to the original settings. Specifically, for tasks such as role-playing, writing skills, and open-ended questions, the generation temperature was set to 0.7; for other tasks, it was set to 0.1 to ensure the generated results were sufficiently diverse yet stable.

In the AlignBench table, DeepSeek 67B Chat DPO scored 6.69 overall, above the listed ChatGPT score of 6.08 and below the two GPT-4 versions at 8.01 and 7.53. Its total score rose from 6.43 to 6.69 after DPO, but its logical-reasoning score fell from 5.79 to 5.41. We cannot say every category improved.

Looking more closely at the columns, the DPO model scored 7.47 for Chinese comprehension, above the listed GPT-4-1106-preview score of 7.33; on basic tasks, its 7.29 was below that model’s 7.99. Its math and logical-reasoning scores were above most Chinese models in the table, but below both GPT-4 versions.

English Open-Ended Evaluation

Paper Table 8: MT-Bench category and average scores.

In the English open-ended evaluation, the research team used MT-Bench, which includes 8 categories of multi-turn dialogue questions to compare responses to these types of prompts.

DeepSeek LLM 67B Chat averaged 8.35, above the listed LLaMA-2-Chat 70B, Xwin 70b v0.1, and TÜLU 2+DPO 70B scores, and close to GPT-3.5-turbo-0613 at 8.39. That comparison is specific to this benchmark and these listed models.

After DPO, its average rose from 8.35 to 8.76, below the listed GPT-4 score of 9.26. The Coding column, however, fell from 7.35 to 6.75.

Other Interesting Findings

After completing the full evaluation process, the research team made several interesting discoveries:

  • (1) Model Safety Evaluation:

    Paper Table 11: Do-Not-Answer benchmark scores; some comparator values come from the benchmark paper.

    The research team used the "Do-Not-Answer Score" to evaluate responses to 939 risky prompts. A higher score means passing more tests under this dataset and scoring procedure.

    In the table, DeepSeek-67B-Chat scored 97.8, above ChatGPT at 97.7 and GPT-4 at 96.5, but below Claude at 98.3 and LLaMA-2-7B-Chat at 99.4. The DeepSeek value was evaluated by the paper authors; the comparator scores came from the original benchmark paper.

  • (2) Staged Fine-Tuning Strategy:

    During the fine-tuning process, the research team discovered that for smaller models (such as 7B), extended fine-tuning on mathematical and programming data can enhance their expertise, but it may harm dialogue performance, causing repetitive responses.

    To address this, they proposed a staged fine-tuning strategy: the first stage uses all available data, while the second focuses on dialogue data. In the 7B experiment, HumanEval stayed at 48.2 and GSM8K dipped from 63.9 to 63.0 after stage two; the repetition rate fell from 2.0% to 1.4%, while IFEval rose from 38.0 to 41.2. These numbers show the trade-off in this experiment.

  • (3) Impact of Multiple-Choice Question Data:

    During the alignment phase, the research team attempted to add 20 million Chinese multiple-choice question data to improve the model’s performance on multiple-choice tasks.

    After adding multiple-choice data, MMLU, C-Eval, and CMMLU scores rose. TriviaQA stayed at 57.9, while Chinese question answering fell from 75.0 to 74.4. The gains on multiple-choice benchmarks did not carry over to these two generative evaluations.

    Therefore, to avoid the model overfitting to the multiple-choice question format and diminishing its overall intelligence performance, they ultimately decided not to include multiple-choice data in both the pre-training and fine-tuning stages.

  • (4) Instruction Data in Pre-Training Stage:

    Additionally, the research team experimented by adding 5 million multiple-choice instruction data points in the later stages of pre-training (the last 10%) to observe the impact on the base model's performance.

    The results showed that this did indeed improve the model’s performance on benchmark tasks, but the final effect was nearly identical to adding the same data during the supervised fine-tuning stage.

    Thus, if the instruction data is large enough, including it in pre-training is feasible. However, since the research team preferred to exclude multiple-choice data and non-multiple-choice data was relatively limited, they ultimately decided not to include instruction data in the pre-training stage.

  • (5) Impact of System Prompts:

    System prompts are used to guide the model in generating helpful, polite, and positive responses.

    The research team used system prompts similar to those in LLaMA-2, such as clearly telling the model, “You are DeepSeek Chat, a helpful, respectful, and honest AI assistant,” and explaining the knowledge cutoff date, etc.

    In the table, adding a system prompt lowered the 7B model’s MT-Bench score from 7.15 to 7.11, while raising the 67B model’s score from 8.35 to 8.58. The authors suggest model ability and a training/test mismatch as possible explanations, but the table does not isolate either cause.

Conclusion

DeepSeek LLM is a large-scale language model trained from the ground up, built on a corpus of 2 trillion tokens in both Chinese and English. The paper not only records the choice of hyperparameters and Scaling Laws calibration during the model training process but also compares various fine-tuning strategies in depth and proposes a methodology that balances training efficiency and resource allocation.

This release is not just about the model itself but also marks an important milestone as DeepSeek-AI officially enters the open-source LLM space.

Currently, the DeepSeek series has released several core papers. We will take the time to read the subsequent research papers and continue to explore its full technical layout and evolution trajectory.