LLM-Assisted Code Cleaning For Training Accurate Code Generators

By Naman Jain, Tianjun Zhang, Wei-Lin Chiang, Joseph E. Gonzalez, Koushik Sen, Ion Stoica. 2023

54 citations recorded by the source corpus.

Natural language to code generation is an important application area of LLMs and has received wide attention from the community. The majority of relevant studies have exclusively concentrated on increasing the quantity and functional correctness of training sets while disregarding other stylistic elements of programs. More recently, data quality has garnered a lot of interest and multiple works have showcased its importance for improving performance. In this work, we investigate data quality for code and find that making the code more structured and readable leads to improved code generation performance of the system. We build a novel data-cleaning pipeline that uses these principles to transform existing programs by 1.) renaming variables, 2.) modularizing and decomposing complex code into smaller helper sub-functions, and 3.) inserting natural-language based plans via LLM based transformations. We evaluate our approach on two challenging algorithmic code generation benchmarks and find that fine-tuning CodeLLaMa-7B on our transformed modularized programs improves the performance by up to 30% compared to fine-tuning on the original dataset. Additionally, we demonstrate improved performance from using a smaller amount of higher-quality data, finding that a model fine-tuned on the entire original dataset is outperformed by a model trained on 15% of our cleaned dataset. Even in comparison to closed-source models, our models outperform the much larger AlphaCoder models.

Subfields: Artificial Intelligence, Information Systems, Software, Large Language Models, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tianyi Zhou
2. Can Xu
3. Armando Solar-Lezama
4. Tao Shen
5. Dacheng Tao
6. Koushik Sen
7. Naman Jain
8. Tianjun Zhang

In Information Systems

1. Armando Solar-Lezama
2. Dacheng Tao
3. Koushik Sen
4. Naman Jain
5. Tianyi Zhou
6. Tianjun Zhang
7. Ion Stoica
8. Leandro Von Werra

In Software

1. Armando Solar-Lezama
2. Koushik Sen
3. Naman Jain
4. Tianjun Zhang
5. Ion Stoica
6. Patrice Godefroid
7. Rishabh Singh
8. Gordon Fraser

In Large Language Models

1. Armando Solar-Lezama
2. Dacheng Tao
3. Naman Jain
4. Tianjun Zhang
5. Niklas Muennighoff
6. Ion Stoica
7. Can Xu
8. Jean Kaddour

In Code Generation & Program Synthesis

1. Armando Solar-Lezama
2. Koushik Sen
3. Rishabh Singh

Citations in the corpus

• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, background): “…as AlphaCode (Li et al., 2022), AlphaCode 2(Gemini Team et al., 2023), ALGO (Zhang et al., 2023d), Parsel (Zelikman et al., 2022), code cleaning (Jain et al., 2023), code explainations (Li et al., 2023a), analogical reasoning (Yasunaga et al., 2023), and AlphaCodium (Ridnik et al., 2024) have…”
• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (2024, background): “While there have been attempts to improve the data quality with LLMs [28], LLMs have their own bias and cannot reliably perform refinement [70].”
• A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “…successfully distilled code knowledge from LLMs into those compact and specialized code models (Chaudhary, 2023; Rozi`ere et al., 2023; Gu-nasekar et al., 2023; Wei et al., 2023; Chen et al., 2023a; Liu et al., 2023e; Yu et al., 2024; Jain et al., 2023; Su and McMillan, 2023; Guo et al., 2023d).”