DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence
By Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, Y. K. Li and 3 more. 2024
1,786 citations recorded by the source corpus.
The rapid development of large language models has revolutionized code intelligence in software development. However, the predominance of closed-source models has restricted extensive research and development. To address this, we introduce the DeepSeek-Coder series, a range of open-source code models with sizes from 1.3B to 33B, trained from scratch on 2 trillion tokens. These models are pre-trained on a high-quality project-level code corpus and employ a fill-in-the-blank task with a 16K window to enhance code generation and infilling. Our extensive evaluations demonstrate that DeepSeek-Coder not only achieves state-of-the-art performance among open-source code models across multiple benchmarks but also surpasses existing closed-source models like Codex and GPT-3.5. Furthermore, DeepSeek-Coder models are under a permissive license that allows for both research and unrestricted commercial use.
Subfields: Artificial Intelligence, Information Systems, Large Language Models, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Percy Liang
- Mohit Bansal
- Richard Socher
- Yoshua Bengio
- Quoc V. Le
- Luke S. Zettlemoyer
- Caiming Xiong
In Information Systems
- Chun Xia
- Lingming Zhang
- Yinlin Deng
- Ling-Ming Zhang
- Soren Dunn
- Armando Solar-Lezama
- Fei Huang
- Zhihong Shao
In Large Language Models
- Richard Socher
- Jianfeng Gao
- Lingming Zhang
- Chun Xia
- Armando Solar-Lezama
- Fei Huang
- Peiyi Wang
- Zhihong Shao
In Code Generation & Program Synthesis
Citations in the corpus
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, background): “…models (Chen et al., 2021; Austin et al., 2021; Li et al., 2022; Zhong et al., 2022; Allal et al., 2023; Li et al., 2023b; Roziere et al., 2023; Guo et al., 2024; Luo et al., 2023; Royzen et al., 2023; Wei et al., 2023b; Ridnik et al., 2024; Lozhkov et al., 2024) and their applications across…”
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “5 7B (Guo et al., 2024), as we notice that starting from a code training model is a better choice compared to a general LLM.”
- Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning (2025, neutral): “Despite demonstrating remarkable reasoning (Guo et al., 2025) and coding (Guo et al., 2024) capabilities, LLMs (Zhao et al., 2023; Team, 2024; Achiam et al., 2023) often lack domain-specific knowledge (Peng et al., 2023; Li et al., 2023) and are prone to hallucinations (Zhang et al., 2023).”
- LoRA Learns Less and Forgets Less (2024, methodology): “In code, models such as StarCoder (Li et al., 2023a; Lozhkov et al., 2024), DeepSeek Coder (Guo et al., 2024), and SantaCoder (Allal et al., 2023) were pretrained from scratch on large-scale code datasets.”
- OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement (2024, neutral): “7B and DeepSeekCoder-Base-33B.”
- Qwen2.5-Coder Technical Report (2024, methodology): “5-Coder-7B-Base stands as the leading model among those over 6 billion parameters, matching the performance of the formidable 33 billion parameter model, DS-Coder-33B-Base.”
- Mixture-of-Agents Enhances Large Language Model Capabilities (2024, background): “For instance, some models excel at complex instruction following (Xu et al., 2023a) while others may be better suited for code generation (Roziere et al., 2023; Guo et al., 2024).”
- Large Language Models: A Survey (2024, positive): “…[103], MeTaLM [104], Dromedary [105], Palmyra [106], Camel [107], Yalm [108], MPT [109], ORCA-2 [110], Gorilla [67], PAL [111], Claude [112], CodeGen 2 [113], Zephyr [114], Grok [115], Qwen [116], Mamba [30], Mixtral-8x7B [117], DocLLM [118], DeepSeek-Coder [119], FuseLLM-7B [120], TinyLlama-1.”
- Agentless: Demystifying LLM-based Software Engineering Agents (2024, methodology): “We recursively traverse the entire repository to obtain the structure, which will be used as input for the LLM.”
- DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence (2024, positive): “The open-source community has made significant strides in advancing code intelligence through the development of open-source code models such as StarCoder (Li et al., 2023b; Lozhkov et al., 2024), CodeLlama (Roziere et al., 2023), DeepSeek-Coder (Guo et al., 2024), and Codestral (MistralAI, 2024).”
- A Survey on Large Language Models for Code Generation (2024, neutral): “These four parts are supported by 38 references (including research papers and technical articles).”
- Process Reinforcement through Implicit Rewards (2025, neutral): “…on competition-level mathematics and programming tasks, including AIME 2024 (Li et al., 2024), AMC (Li et al., 2024), MATH-500 (Hendrycks et al., 2021b), Minerva Math (Lewkowycz et al., 2022), Olympiad-Bench (He et al., 2024), LeetCode (Guo et al., 2024), and LiveCodeBench (v2) (Jain et al., 2024).”
- Syntriever: How to Train Your Retriever with Synthetic Data from LLMs (2025, unknown): “Large Language Models (LLMs) have become a core technology in various NLP applications such as chatbots (Achiam et al., 2023; Team et al., 2023) and coding assistants (Roziere et al., 2023; Guo et al., 2024).”
- Across Programming Language Silos: A Study on Cross-Lingual Retrieval-augmented Code Generation (2025, unknown): “7B-Instruct [47], Qwen2.”