Few-shot training LLMs for project-specific code-summarization

By Toufique Ahmed, Prem Devanbu. International Conference on Automated Software Engineering, 2022

359 citations recorded by the source corpus.

Very large language models (LLMs), such as GPT-3 and Codex have achieved state-of-the-art performance on several natural-language tasks, and show great promise also for code. A particularly exciting aspect of LLMs is their knack for few-shot and zero-shot learning: they can learn to perform a task with very few examples. Few-shotting has particular synergies in software engineering, where there are a lot of phenomena (identifier names, APIs, terminology, coding patterns) that are known to be highly project-specific. However, project-specific data can be quite limited, especially early in the history of a project; thus the few-shot learning capacity of LLMs might be very relevant. In this paper, we investigate the use few-shot training with the very large GPT (Generative Pre-trained Transformer) Codex model, and find evidence suggesting that one can significantly surpass state-of-the-art models for code-summarization, leveraging project-specific training.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/53661ff6fdbfb8557c5b19895fad151792c62da7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chun Xia
2. Mingwei Liu
3. Yixuan Chen
4. Jiawei Liu
5. Lingming Zhang
6. Yuyao Wang
7. Kaixin Wang

In Information Systems

1. Chun Xia
2. Lingming Zhang
3. Jiawei Liu
4. Yuyao Wang
5. Yinlin Deng
6. Soren Dunn

Citations in the corpus

• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (2023, background): “if month in [1,3,5,7,8,10,12] and day < 1 or day > 31: return False if month in [4,6,9,11] and day < 1 or day > 30: return False .”
• CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2023, background): “For evaluation, most of works (Ahmed and Devanbu, 2022; Lu et al., 2021; Zhu et al., 2022) still rely on n-gram matching like BLEU (Papineni et al., 2002) and CodeBLEU (Ren et al., 2020), which suffer from the heavy dependence on comprehensiveness and accuracy of reference code.”
• Evaluating Large Language Models in Class-Level Code Generation (2024, methodology): “These include code generation [46, 54, 62, 72], code summarization [12, 51] and various software maintenance tasks, including vulnerability detection [58, 61, 73], test generation [68, 69], and program repair [26, 36, 37, 63].”