DeepSeek LLM: Scaling Open-Source Language Models with Longtermism

By DeepSeek-AI Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, C. Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu and 76 more. 2024

805 citations recorded by the source corpus.

The rapid development of open-source large language models (LLMs) has been truly remarkable. However, the scaling law described in previous literature presents varying conclusions, which casts a dark cloud over scaling LLMs. We delve into the study of scaling laws and present our distinctive findings that facilitate scaling of large scale models in two commonly used open-source configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek LLM, a project dedicated to advancing open-source language models with a long-term perspective. To support the pre-training phase, we have developed a dataset that currently consists of 2 trillion tokens and is continuously expanding. We further conduct supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the creation of DeepSeek Chat models. Our evaluation results demonstrate that DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in the domains of code, mathematics, and reasoning. Furthermore, open-ended evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance compared to GPT-3.5.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Han
2. Maosong Sun
3. Ning Ding
4. Zhiyuan Liu
5. Shengding Hu
6. Dahua Lin
7. Ping Luo
8. Yuan Yao

In Large Language Models

1. Yuan Yao
2. Shengding Hu
3. Maosong Sun
4. Dahua Lin
5. Shoufa Chen
6. Xiao-wen Dong
7. Ge Zhang
8. Yi Jiang

Citations in the corpus

• Muon is Scalable for LLM Training (2025, neutral): “(Kaplan et al. 2020; Hoffmann et al. 2022; Bi et al. 2024) Table 9 presents the results of our systematic parameter search process.”
• WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, positive): “2) By employing diverse pre-trained models (i.e., GPT-2, Llama 2, Mistral, Qwen, DeepSeek) as base models, WizardMath demonstrated notable advancements on the GSM8k and MATH benchmarks.”
• Scaling Synthetic Data Creation with 1,000,000,000 Personas (2024, background): “Similar to math problems, logical reasoning problems can also be easily synthesized.”
• Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation (2025, positive): “As evidenced in other fields [115, 4, 3], the prosperity of a domain in the era of large models usually relies on a strong open-source foundational model, such as Stable Diffusion [74, 69, 24] for image generation, LLaMA [91, 92, 22] for language models, and HunyuanVideo [43] for video generation.”
• How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “Large language models (LLMs) [3, 6, 9, 10, 13, 26, 99, 101, 103, 107, 108, 117, 118, 136] have greatly advanced AGI by enabling complex language tasks previously thought human-exclusive.”
• Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation (2024, background): “…2022; OpenAI 2022, 2023b; Google 2023; Anthropic 2023; Workshop et al. 2022; Touvron et al. 2023a,b; Bai et al. 2023a; Yang et al. 2023; Team 2023; Bi et al. 2024] and incredible scalability [Kaplan et al. 2020; Henighan et al. 2020; Hoffmann et al. 2022; Wei et al. 2022; Alabdulmohsin et al.…”
• Are We on the Right Way for Evaluating Large Vision-Language Models? (2024, methodology): “Specifically, we employ two closed-source LLMs [41, 34] and six open-source LLMs [1, 42, 49, 3, 19, 43] sizing 20B or larger to serve as inspectors.”
• CodeMind: A Framework to Challenge Large Language Models for Code Reasoning (2024, neutral): “…(Touvron et al., 2023), Mistral (Jiang et al., 2023), CodeLlama (13B, instruction-tuned)(Roziere et al., 2023), StarCoder (15.5B)(Li et al., 2023), Wizard-Coder (15B, instruction-tuned)(Xu et al., 2023), Magic-Coder (7B)(Wei et al., 2023) (instruction-tuned), DeepSeek-Coder (6.7B)(Bi et al., 2024).”
• A Survey on Mixture of Experts in Large Language Models (2024, neutral): “Specifically, DeepSeekMoE-145B employs a reduced intermediate hidden dimension at one-eighth that of its dense FFN counterpart, increasing both the number of experts (from 16 to 128) and the number of active experts (from top-2 to top-16) by a factor of eight.”
• SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (2024, neutral): “Benefiting from tremendous advancements in large language models (LLMs) (Radford et al., 2019; Brown et al., 2020; Achiam et al., 2023; Touvron et al., 2023; Peng et al., 2023; Bi et al., 2024), the realm of vision-language models (VLMs) has undergone a revolutionary progress.”
• Yi: Open Foundation Models by 01.AI (2024, methodology): “Additionally, research conducted by DeepSeek-AI et al. [17] has highlighted that the allocation of an increased computational budget towards model scaling should be proportional to the quality of the data available.”
• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (2024, background): “Of these, DeepSeek (Bi et al., 2024) bears the closest resemblance to our proposed WSD LRS.”
• Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, neutral): “3 B) [5] with a maximum supported sequence length of 4096 as the base language model.”
• Large Language Diffusion Models (2025, neutral): “To comprehensively evaluate the in-context learning and instruction-following capabilities of LLaDA 8B, we conducted detailed comparisons with existing LLMs [6, 21, 25, 26, 32, 33] of similar scale.”