SantaCoder: don't reach for the stars!

By Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Muñoz Ferrandis, Niklas Muennighoff, Mayank Mishra, A. Gu, Manan Dey and 31 more. 2023

247 citations recorded by the source corpus.

The BigCode project is an open-scientific collaboration working on the responsible development of large language models for code. This tech report describes the progress of the collaboration until December 2022, outlining the current state of the Personally Identifiable Information (PII) redaction pipeline, the experiments conducted to de-risk the model architecture, and the experiments investigating better preprocessing methods for the training data. We train 1.1B parameter models on the Java, JavaScript, and Python subsets of The Stack and evaluate them on the MultiPL-E text-to-code benchmark. We find that more aggressive filtering of near-duplicates can further boost performance and, surprisingly, that selecting files from repositories with 5+ GitHub stars deteriorates performance significantly. Our best model outperforms previous open-source multilingual code generation models (InCoder-6.7B and CodeGen-Multi-2.7B) in both left-to-right generation and infilling on the Java, JavaScript, and Python portions of MultiPL-E, despite being a substantially smaller model. All models are released under an OpenRAIL license at https://hf.co/bigcode.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Code Generation & Program Synthesis

Citations in the corpus

Code Llama: Open Foundation Models for Code (2023, background): “In addition to program synthesis from natural language prompts or infilling (Fried et al., 2023; Bavarian et al., 2022; Li et al., 2023; Nguyen et al., 2023), many tasks related to code understanding or synthesis have been addressed since the early 2020s with NLP models adapted for code (Raffel et a”
StarCoder: may the source be with you! (2023, methodology): “Most of these models have not been fully open, but PolyCoder (Xu et al., 2022) and SantaCoder (Ben Allal et al., 2023) are notable exceptions and have both open models and training data.”
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023, methodology): “However, MultiPL-E-generated solutions are used as multi-language inﬁlling benchmarks for the SantaCoder [33] code generation models, which do support in-ﬁlling.”
Execution-Based Evaluation for Open-Domain Code Generation (2022, background): “Table 20 shows the effect when using different numbers of test cases for execution-based evaluation.”
CodeGen2: Lessons for Training LLMs on Programming and Natural Languages (2023, background): “In recent work (Chen et al., 2021b; Nijkamp et al., 2022; Fried et al., 2022; Allal et al., 2023), these ﬁndings have been transferred from natural to programming languages and achieved impressive performance in program synthesis and understanding tasks.”
RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems (2023, background): “More recent LLMs, including Codex [9], PolyCoder [46], CodeGen [27], In-Coder [15], CodeGeeX [49], SantaCoder [2], and StarCoder [23], employ billions of parameters and excel in code generation tasks, benefiting from large-scale, highquality code corpora.”
RLTF: Reinforcement Learning from Unit Test Feedback (2023, methodology): “…CodeGen (Nijkamp et al., 2022), PaLM-Coder (Chowdhery et al., 2022), PanGu-Coder (Christopoulou et al., 2022), CodeGeex (Zheng et al., 2023), and SantaCoder (Allal et al., 2023), as well as encoder-decoder transformer architectures like PLABRT (Ahmad et al., 2021) and CodeT5 (Wang et al., 2021).”
LoRA Learns Less and Forgets Less (2024, methodology): “In code, models such as StarCoder (Li et al., 2023a; Lozhkov et al., 2024), DeepSeek Coder (Guo et al., 2024), and SantaCoder (Allal et al., 2023) were pretrained from scratch on large-scale code datasets.”
Measuring The Impact Of Programming Language Distribution (2023, background): “, 2022), and other works (Nijkamp et al., 2022; Fried et al., 2022; Allal et al., 2023; Christopoulou et al., 2022) have shown that decoder-only code language models achieve exceptional performance on a wide range of tasks.”
Can Transformers Learn to Solve Problems Recursively? (2023, background): “Even models trained on code often need to be finetuned to solve specific tasks instead of used in a multitask fashion [4, 2, 20].”
Exploring AI Ethics of ChatGPT: A Diagnostic Analysis (2023, background): “One ethical practice is to filter the personally identifiable information, which has been adopted by some recent LLMs [73, 74, 75].”
C-Pack: Packaged Resources To Advance General Chinese Embedding (2023, background): “…2021b,a) in line with observations for the importance of scaling LLMs (Hoffmann et al., 2022; Rae et al., 2021; Brown et al., 2020; Chowdhery et al., 2022; Srivastava et al., 2022; Gao et al., 2021a; Li et al., 2023a; Allal et al., 2023; Muennighoff et al., 2023b).”
DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence (2024, methodology): “In previous works (Chen et al., 2021; Li et al., 2023; Nijkamp et al., 2022; Roziere et al., 2023), large language models for code are mainly pre-trained on file-level source code, which ignores the dependencies between different files in a project.”
Qwen Technical Report (2023, unknown): “, 2022), SantaCoder (Allal et al., 2023), Starcoder-Base (Li et al.”