DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

By Zhihong Shao, Damai Dai, Daya Guo, Zihan Wang, Huajian Xin. 2024

1,318 citations recorded by the source corpus.

We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-quality and multi-source corpus consisting of 8.1T tokens, and further perform Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to fully unlock its potential. Evaluation results show that, even with only 21B activated parameters, DeepSeek-V2 and its chat versions still achieve top-tier performance among open-source models.

Subfields: Artificial Intelligence, Information Systems, Large Language Models, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xueguang Ma
2. Ge Zhang
3. Azalia Mirhoseini
4. Xiang Yue
5. Ziyan Jiang
6. Wenhu Chen
7. Quoc V. Le
8. Mohit Bansal

In Information Systems

1. Xueguang Ma
2. Quoc V. Le

In Large Language Models

1. Ge Zhang
2. Xueguang Ma
3. Quoc V. Le
4. Jason Aaron Edward Weston
5. Jianfeng Gao
6. Kyunghyun Cho
7. Jiwei Li
8. Verena Rieser

In Model Architectures

1. Azalia Mirhoseini
2. Ziyan Jiang
3. Wenhu Chen
4. Quoc V. Le
5. Yoshua Bengio
6. Barret Zoph
7. Kyunghyun Cho
8. Frank Hutter

Citations in the corpus

• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (2024, methodology): “Advanced open-source models like Llama-3-70B-Instruct [24] and DeepSeek-V2-Chat [15], while not yet performing at the level of leading closed-source models such as GPT-4o and Claude-3-Opus, have shown performance that is close to Claude-3-Sonnet.”
• MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (2024, neutral): “Large language models (LLMs) have entered the era of long-context processing, with some of them supporting context windows ranging from 128K to 10M tokens [Gra24, RST + 24, LYZA24, YCL + 24, AJA + 24, DA24].”
• Large Language Monkeys: Scaling Inference Compute with Repeated Sampling (2024, positive): “For SWE-bench Lite, we use DeepSeek-Coder-V2-Instruct [20], as the required context length of this task exceeds the limits of the Llama-3 models.”
• Kimi K2: Open Agentic Intelligence (2025, neutral): “The architecture follows a similar design to DeepSeek-V3 [11] , employing Multi-head Latent Attention (MLA) [45] as the attention mechanism, with a model hidden dimension of 7168 and an MoE expert hidden dimension of 2048.”
• A Survey on Mixture of Experts in Large Language Models (2024, neutral): “Switch Transformers [34] and many other subsequent studies [26], [28], [33], [66] have embraced this L aux design, and enhancements [30], [65], [67] have been made to cater to diverse requirements.”
• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (2024, neutral): “We repeat the benchmarks 10 times and take the average timing.”