StarCoder 2 and The Stack v2: The Next Generation

By Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, J. Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei and 60 more. arXiv.org, 2024

698 citations recorded by the source corpus.

The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership with Software Heritage (SWH), we build The Stack v2 on top of the digital commons of their source code archive. Alongside the SWH repositories spanning 619 programming languages, we carefully select other high-quality data sources, such as GitHub pull requests, Kaggle notebooks, and code documentation. This results in a training set that is 4x larger than the first StarCoder dataset. We train StarCoder2 models with 3B, 7B, and 15B parameters on 3.3 to 4.3 trillion tokens and thoroughly evaluate them on a comprehensive set of Code LLM benchmarks. We find that our small model, StarCoder2-3B, outperforms other Code LLMs of similar size on most benchmarks, and also outperforms StarCoderBase-15B. Our large model, StarCoder2- 15B, significantly outperforms other models of comparable size. In addition, it matches or outperforms CodeLlama-34B, a model more than twice its size. Although DeepSeekCoder- 33B is the best-performing model at code completion for high-resource languages, we find that StarCoder2-15B outperforms it on math and code reasoning benchmarks, as well as several low-resource languages. We make the model weights available under an OpenRAIL license and ensure full transparency regarding the training data by releasing the SoftWare Heritage persistent IDentifiers (SWHIDs) of the source code data.

View Paper: https://www.semanticscholar.org/paper/18e7ab056c16928d8f9539509a4b366889106d97

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. An Yang
2. Terry Yue Zhuo
3. Binyuan Hui
4. Naman Jain
5. Wen-Ding Li
6. J. Portes
7. Minh Chien Vu
8. Jenny Chim

Citations in the corpus

• LoRA Learns Less and Forgets Less (2024, methodology): “In code, models such as StarCoder (Li et al., 2023a; Lozhkov et al., 2024), DeepSeek Coder (Guo et al., 2024), and SantaCoder (Allal et al., 2023) were pretrained from scratch on large-scale code datasets.”
• Training Language Models to Self-Correct via Reinforcement Learning (2024, unknown): “Large language models (LLMs) are a useful tool for reasoning in scientific domains such as math and coding (Shao et al., 2024; Lozhkov et al., 2024; Team, 2024).”
• Kimi k1.5: Scaling Reinforcement Learning with LLMs (2025, unknown): “5 model is trained in three stages: the vision-language pretraining stage, the vision-language cooldown stage, and the long-context activation stage.”
• DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence (2024, unknown): “These models are available in four sizes: 7B, 13B, 34B, and 70B. • StarCoder (Lozhkov et al., 2024) is a publicly accessible model with 15 billion parameters.”
• A Survey on Large Language Models for Code Generation (2024, unknown): “Then, this paper compares these four techniques with ICL and traditional full fine-tuning on code generation tasks using Python datasets like CoNaLa or CodeAlpaca.”
• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (2024, methodology): “More recently, pre-trained base code models [41, 34, 36, 52] have been built to perform accurate code completion.”
• Qwen2.5-Coder Technical Report (2024, methodology): “Built upon pretrained LLMs, code LLMs such as the StarCoder series (Li et al., 2023; Lozhkov et al., 2024), CodeLlama series (Roziere et al., 2023), DeepSeek-Coder series (Guo et al., 2024), CodeQwen1.”
• Enhancing Cross-Language Code Translation via Task-Specific Embedding Alignment in Retrieval-Augmented Generation (2024, unknown): “We collect a dataset of 25,000 Fortran code examples from Stack V2 (Lozhkov et al., 2024) and use the LLaMA 3.1-8B (Touvron et al., 2023) model to generate corresponding C++ translations.”
• Enhancing Code Translation in Language Models with Few-Shot Learning via Retrieval-Augmented Generation (2024, unknown): “3) Stack-V2 Dataset:: The Stack V2 dataset is a comprehensive collection of code samples sourced from a wide range of repositories on GitHub, focusing on high-performance computing and various computational problems [21].”
• VeriCoder: Enhancing LLM-Based RTL Code Generation through Functional Correctness Validation (2025, unknown): “However, existing open-source models still underperform on RTL tasks, largely due to the absence of high-quality, functionally validated RTL datasets in their training corpora [21], [22].”