Code Llama: Open Foundation Models for Code
By Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Tan, Yossi Adi, Jingyu Liu, Tal Remez, J. Rapin and 15 more. 2023
3,394 citations recorded by the source corpus.
We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B, 34B and 70B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B, 13B and 70B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 67% and 65% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.
Subfields: Artificial Intelligence, Computer Networks and Communications, Software, Large Language Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Graham Neubig
- Furu Wei
- Quoc V. Le
- Yoshua Bengio
- Percy Liang
In Computer Networks and Communications
- Ion Stoica
- Joseph E. Gonzalez
- Yinmin Zhong
- Yiming Yang
- Song Han
- Prateek Mittal
- Stella Biderman
- Huan Sun
In Software
In Large Language Models
- Furu Wei
- Jianfeng Gao
- Graham Neubig
- Richard Socher
- Nan Duan
- Marie-Anne Lachaux
- Maosong Sun
- Wenhu Chen
Citations in the corpus
- Mistral 7B (2023, background): “Furthermore, Mistral 7B approaches the coding performance of Code-Llama 7B [20], without sacrificing performance on non-code related benchmarks.”
- Llemma: An Open Language Model For Mathematics (2023, methodology): “Each model is initialized from Code Llama (Rozière et al., 2023).”
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, neutral): “…of code-specific models (Chen et al., 2021; Austin et al., 2021; Li et al., 2022; Zhong et al., 2022; Allal et al., 2023; Li et al., 2023b; Roziere et al., 2023; Guo et al., 2024; Luo et al., 2023; Royzen et al., 2023; Wei et al., 2023b; Ridnik et al., 2024; Lozhkov et al., 2024) and…”
- A Survey on Large Language Models for Code Generation (2024, neutral): “…for Optimization [40] Parameter-Efficient Fine-Tuning [41] Data Pruning for Fine-Tuning [42] B. Feedback ClarifyGPT Framework [43] RLEF [44] Crowd-Sourced RLHF [45] C. Prompt Engineering Chain-of-Thought Prompting [46] AceCoder Technique [47] Security-Focused Prompting [48] V. Evaluations”
- CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2023, neutral): “Existing evaluation methods are significantly insufficient in terms of assessing integrity and comprehensiveness for code understanding and generation capabilities of LLMs. Firstly, many code LLMs, such as CodeT5+ (Wang et al., 2023b), WizardCoder (Luo et al., 2023), and Code LLaMA (Rozière et al., ”
- NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness (2024, background): “Among open-weights models, Phind-CodeLlama-34B results in the highest overall performance across all the NoFunEdit tasks, almost consistently outperforming DeepSeekCoder-33B-Inst .”
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (2024, methodology): “Specifically, following prior works [52, 35, 32], we report Pass@1 with greedy decoding for the main experiments in the zero-shot setting.”
- Continual Learning of Large Language Models: A Comprehensive Survey (2024, neutral): “Continual DAP which involves more than one stage is seldom explored: among all articles listed in Table 2, only one employs two stages of DAP (“PT → DAP → DAP → FT” in Code Llama [199]).”
- Large Language Models: A Survey (2024, positive): “The LLaMA family is growing rapidly, as more instruction-following models have been built on LLaMA or LLaMA-2, including Code LLaMA [66], Gorilla [67], Giraffe [68], Fig.”
- PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling (2024, background): “Large language models (LLMs) [1, 27, 28, 18] are integral to various natural language processing applications, including dialogue systems [5], document summarization [9], and code completion [25].”
- Agentless: Demystifying LLM-based Software Engineering Agents (2024, methodology): “The repository structure format provides the necessary file paths alongside the neighboring file names to maintain organizational information in the original codebase.”
- ChipNeMo: Domain-Adapted LLMs for Chip Design (2023, methodology): “LLMs have proven adept at small scale code generation on a wide array of tasks [32] and therefore customizing these models to accelerate engineer productivity in this domain specific task is a natural fit.”
- Qwen2.5-Coder Technical Report (2024, methodology): “Built upon pretrained LLMs, code LLMs such as the StarCoder series (Li et al., 2023; Lozhkov et al., 2024), CodeLlama series (Roziere et al., 2023), DeepSeek-Coder series (Guo et al., 2024), CodeQwen1.”
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023, methodology): “Following precedent in code generation for evaluation in Pass@1 (Chen et al., 2021; Rozière et al., 2023), we simply use greedy decoding for all models.”
- Effective Long-Context Scaling of Foundation Models (2023, background): “The idea is also concurrently suggested in the Red-dit (r/LocalLLaMa) community and Rozière et al. (2023).”
- DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence (2024, background): “In previous works (Chen et al., 2021; Li et al., 2023; Nijkamp et al., 2022; Roziere et al., 2023), large language models for code are mainly pre-trained on file-level source code, which ignores the dependencies between different files in a project.”
- Efficient Streaming Language Models with Attention Sinks (2023, background): “…language processing applications such as dialog systems (Schulman et al., 2022; Taori et al., 2023; Chiang et al., 2023), document summarization (Goyal & Durrett, 2020; Zhang et al., 2023a), code completion (Chen et al., 2021; Rozière et al., 2023) and question answering (Kamalloo et al., 2023).”