CodeGen2: Lessons for Training LLMs on Programming and Natural Languages

By Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, Yingbo Zhou. 2023

243 citations recorded by the source corpus.

Large language models (LLMs) have demonstrated remarkable abilities in representation learning for program synthesis and understanding tasks. The quality of the learned representations appears to be dictated by the neural scaling laws as a function of the number of model parameters and observations, while imposing upper bounds on the model performance by the amount of available data and compute, which is costly. In this study, we attempt to render the training of LLMs for program synthesis more efficient by unifying four key components: (1) model architectures, (2) learning methods, (3) infill sampling, and, (4) data distributions. Specifically, for the model architecture, we attempt to unify encoder and decoder-based models into a single prefix-LM. For learning methods, (i) causal language modeling, (ii) span corruption, (iii) infilling are unified into a simple learning algorithm. For infill sampling, we explore the claim of a"free lunch"hypothesis. For data distributions, the effect of a mixture distribution and multi-epoch training of programming and natural languages on model performance is explored. We conduct a comprehensive series of empirical experiments on 1B LLMs, for which failures and successes of this exploration are distilled into five lessons. We will provide a final recipe for training and release CodeGen2 models in size 1B, 3.7B, 7B, and, 16B parameters, along with the training framework as open-source: https://github.com/salesforce/CodeGen.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Information Systems, Large Language Models, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Steven C. H. Hoi

In Information Systems

In Large Language Models

In Code Generation & Program Synthesis

Citations in the corpus

Code Llama: Open Foundation Models for Code (2023, neutral): “…et al., 2021), CodeT5 (Wang et al., 2021), InCoder (Fried et al., 2023), AlphaCode (Li et al., 2022), CodeGen (Nijkamp et al., 2023b) and CodeGen 2 (Nijkamp et al., 2023a), GPT-NeoX (Black et al., 2022), SantaCoder (Allal et al., 2023), StarCoder (Li et al., 2023) and phi-1 (Gunasekar et al.,…”
NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness (2024, background): “…more capable language models (LMs) of code, e.g., (Chen et al., 2021; Wang & Komatsuzaki, 2021; Austin et al., 2021; Black et al., 2022; Tunstall et al., 2022; Fried et al., 2022; Nijkamp et al., 2023; Li et al., 2023b; Wang et al., 2023; OpenAI, 2023; Luo et al., 2023; Muennighoff et al., 2023).”
WizardCoder: Empowering Code Large Language Models with Evol-Instruct (2023, unknown): “…also extends to the realm of code understanding and generation, where a multitude of Code LLMs have emerged (Chen et al., 2021a; Li et al., 2022; Fried et al., 2022; Nijkamp et al., 2023b; Zheng et al., 2023; Wang et al., 2021; 2023; Li et al., 2023a; Nijkamp et al., 2023a; Rozi`ere et al., 2023).”
Large Language Models: A Survey (2024, unknown): “…[103], MeTaLM [104], Dromedary [105], Palmyra [106], Camel [107], Yalm [108], MPT [109], ORCA-2 [110], Gorilla [67], PAL [111], Claude [112], CodeGen 2 [113], Zephyr [114], Grok [115], Qwen [116], Mamba [30], Mixtral-8x7B [117], DocLLM [118], DeepSeek-Coder [119], FuseLLM-7B [120], TinyLlama-1.”
Gorilla: Large Language Model Connected with Massive APIs (2023, unknown): “Besides prompting, there have also been efforts to pretrain language models specifically for code generation [25, 21, 26].”
CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023, methodology): “However, these decoder-only models are often not ideal for understanding tasks such as retrieval and detection tasks compared to encoder-only models [Nijkamp et al., 2023a].”
Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (2023, background): “co/datasets?other=code-generation (4)To date, CodeGen2-16B is released with an unfinished checkpoint [37].”
CodeS: Towards Building Open-source Language Models for Text-to-SQL (2024, unknown): “This mixed training in natural language and code offers benefits for a wide range of tasks in both domains [45].”