DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

By Damai Dai, C. Deng, Chenggang Zhao, Runxin Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu and 7 more. Annual Meeting of the Association for Computational Linguistics, 2024

972 citations recorded by the source corpus.

In the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top-$K$ out of $N$ experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into $mN$ ones and activating $mK$ from them, allowing for a more flexible combination of activated experts; (2) isolating $K_s$ experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5 times the expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations. Further, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems

View Paper: https://www.semanticscholar.org/paper/16d6e1ed1cf72212f6154644f3aa59d18bc95fda

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Damai Dai
2. Fei Huang
3. Zhihong Shao
4. Chujie Zheng
5. Binyuan Hui
6. Jingren Zhou
7. Sinan Tan
8. Xiaoming Shi

In Computer Vision and Pattern Recognition

1. Jingren Zhou
2. Damai Dai
3. Fei Huang
4. Zhihong Shao
5. Chujie Zheng
6. Sinan Tan
7. Shuai Bai
8. Kai Dang

In Information Systems

1. Fei Huang
2. Zhihong Shao
3. Jingren Zhou
4. Sunghun Kim
5. Daya Guo
6. Juyong Jiang
7. Fan Wang
8. Junyang Lin

Citations in the corpus

• A Survey on Mixture of Experts in Large Language Models (2024, positive): “Switch Transformers [34] and many other subsequent studies [26], [28], [33], [66] have embraced this L aux design, and enhancements [30], [65], [67] have been made to cater to diverse requirements.”
• Qwen2 Technical Report (2024, methodology): “Instruction Evolution To enrich the instruction dataset, a self-evolution strategy (Zhao et al., 2024) is employed, prompting the Qwen models to add constraints or requirements to existing instructions, thereby increasing their complexity and ensuring a diverse range of difficulty levels within the…”
• Qwen3 Technical Report (2025, positive): “5-MoE (Yang et al., 2024b) and implement fine-grained expert segmentation (Dai et al., 2024).”
• Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention (2025, neutral): “Despite these promising strategies, existing sparse attention meth-ods often fall short in practical deployments.”
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (2024, methodology): “Following Dai et al. (2024), we substitute all FFNs except for the first layer with MoE layers.”
• Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts (2024, positive): “To mitigate this, following the approaches of (Dai et al., 2024; Fe-dus et al., 2022), we achieve expert-level balancing with an auxiliary loss to reduce routing collapse: where f i represents the fraction of tokens assigned to expert i , and r i denotes the proportion of router probability…”
• PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, neutral): “5-34B [64] and MoE models: DeepSeek-MoE-16B [65] and Mixtral-8x7B [66].”
• Mixture-of-Experts in Remote Sensing: A Survey (2026, unknown): “For LLMs, architectures like DeepSeekMoE [16], Mixtral [60] and OLMoE [100] position MoE layers inside decoder-only Transformers [139] to obtain large effective capacity with relatively small active parameter counts per token.”
• Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings (2026, unknown): “The 480M and 3B models are dense bidirectional Qwen3 encoders (Zhang et al., 2025), while the largest model is a bidirectional DeepSeekMoE-style encoder (Dai et al., 2024) with 10B total parameters, 64 routed experts, one shared expert, and top-4 routing.”