The Stack: 3 TB of permissively licensed source code

By Denis Kocetkov, Raymond Li, Raymond Li, Loubna Ben Allal, Jia Li, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell and 6 more. 2022

482 citations recorded by the source corpus.

Large Language Models (LLMs) play an ever-increasing role in the field of Artificial Intelligence (AI)--not only for natural language processing but also for code understanding and generation. To stimulate open and responsible research on LLMs for code, we introduce The Stack, a 3.1 TB dataset consisting of permissively licensed source code in 30 programming languages. We describe how we collect the full dataset, construct a permissively licensed subset, present a data governance plan, discuss limitations, and show promising results on text2code benchmarks by training 350M-parameter decoders on different Python subsets. We find that (1) near-deduplicating the data significantly boosts performance across all experiments, and (2) it is possible to match previously reported HumanEval and MBPP performance using only permissively licensed data. We make the dataset available at https://hf.co/BigCode, provide a tool called"Am I in The Stack"(https://hf.co/spaces/bigcode/in-the-stack) for developers to search The Stack for copies of their code, and provide a process for code to be removed from the dataset by following the instructions at https://www.bigcode-project.org/docs/about/the-stack/.

Subfields: Artificial Intelligence, Information Systems, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Evaluation & Benchmarks

Citations in the corpus

Llemma: An Open Language Model For Mathematics (2023, methodology): “Motivated by this fact, we create AlgebraicStack , an 11B-token dataset of source code from 17 languages, spanning numerical, symbolic, and formal math.”
StarCoder: may the source be with you! (2023, methodology): “The assignment of data to programming languages was performed based solely on ﬁle extension (Kocetkov et al., 2022).”
Foundation Models and Fair Use (2023, background): “Similarly, Kocetkov et al. (2022) filter their code dataset for permissive licenses.”
Data Portraits: Recording Foundation Model Training Data (2023, methodology): “Kocetkov et al. [22] gather a set of permissively licensed open source code repositories.”
SantaCoder: don't reach for the stars! (2023, methodology): “In earlier work, the BigCode community released The Stack v1.”
Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models (2022, methodology): “We pre-train GPT-2 on The-Stack dataset (Python only) [83] from BigCode2 and evaluated on the HumanEval dataset [84] by zero-shot learning.”
DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence (2024, unknown): “It is specifically trained on a meticulously curated subset of the Stack dataset (Kocetkov et al., 2022), covering 86 programming languages.”
CodeS: Towards Building Open-source Language Models for Text-to-SQL (2024, unknown): “Specifically, we start by using regular expressions to extract all “ SELECT ” queries from three extensive open-source corpora: The Pile [22], The Stack [31], and GitHub Code 5 .”
Making Large Language Models Better Reasoners with Alignment (2023, unknown): “…improve the reasoning ability of LLMs, which can be broadly divided into three groups: 1) pre-training : The pre-training methods pre-train LLMs on a vast of unsupervised datasets, such as the pile (Gao et al., 2020), the stack (Kocetkov et al., 2022), with a simple next token prediction objective.”
Measuring The Impact Of Programming Language Distribution (2023, background): “Further, despite their similar popularity, Go has nearly 350% more source code available (Kocetkov et al., 2022).”
MathPile: A Billion-Token-Scale Pretraining Corpus for Math (2023, unknown): “The Stack Kocetkov et al. [30] provides a 3.1 TB code dataset.”
CodeGen2: Lessons for Training LLMs on Programming and Natural Languages (2023, methodology): “Examples in a batch are sampled from the Pile (Gao et al., 2020) and the Stack data equally likely.”
Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models (2024, unknown): “We use the 1B, 3B, 7B and 15.5B StarCoderBase models (Li et al., 2023) that were trained on the S TAR C ODER D ATA dataset (Li et al., 2023).”
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore (2023, methodology): “This has been done in prior work, primarily for code-based datasets (e.g., Kocetkov et al., 2023; Fried et al., 2023; Together, 2023) and scientific text (e.g., Soldaini & Lo, 2023).”
Retentive Network: A Successor to Transformer for Large Language Models (2023, background): “The training corpus is a curated compilation of The Pile (Gao et al., 2020), C4 (Dodge et al., 2021), and The Stack (Kocetkov et al., 2022).”
ChipNeMo: Domain-Adapted LLMs for Chip Design (2023, methodology): “For code, we leverage GitHub data [18], focusing on programming languages also present in our internal data chip design dataset such as C++, Python, and Verilog.”
A Comprehensive Overview of Large Language Models (2023, unknown): “2 [348] HumanEval [246], MBPP [345], DS-1000 [349], HELM [350], Multi-Language Evaluation, GSM8K [239], MMLU [221], CoQA [233]”
Scaling Data-Constrained Language Models (2023, methodology): “Figure 6 (left) illustrates the strategies: (a) Code augmentation: We use Python code from The Stack [49] to make up for missing natural language data.”
Textbooks Are All You Need II: phi-1.5 technical report (2023, methodology): “This filtered web data consists of 88B tokens filtered from the Falcon refined web dataset [PMH + 23], and 7B tokens of code data filtered from The Stack [KLA + 22] and StackOverflow.”