LongCoder: A Long-Range Pre-trained Language Model for Code Completion

By Daya Guo, Canwen Xu, Nan Duan, Jian Yin, Julian McAuley. 2023

166 citations recorded by the source corpus.

In this paper, we introduce a new task for code completion that focuses on handling long code input and propose a sparse Transformer model, called LongCoder, to address this task. LongCoder employs a sliding window mechanism for self-attention and introduces two types of globally accessible tokens - bridge tokens and memory tokens - to improve performance and efficiency. Bridge tokens are inserted throughout the input sequence to aggregate local information and facilitate global interaction, while memory tokens are included to highlight important statements that may be invoked later and need to be memorized, such as package imports and definitions of classes, functions, or structures. We conduct experiments on a newly constructed dataset that contains longer code context and the publicly available CodeXGLUE benchmark. Experimental results demonstrate that LongCoder achieves superior performance on code completion tasks compared to previous models while maintaining comparable efficiency in terms of computational resources during inference. All the codes and data are available at https://github.com/microsoft/CodeBERT.

Subfields: Artificial Intelligence, Computer Networks and Communications, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Gabriel Synnaeve
2. Yossi Adi
3. Jade Copet
4. Nicolas Usunier
5. Tal Remez
6. Hugo Touvron
7. Jianfeng Gao
8. Itai Gat

In Computer Networks and Communications

1. Nicolas Usunier
2. Yuxiao Dong
3. Hugo Touvron
4. Wenhan Xiong
5. Baptiste Rozière

In Information Systems

1. Gabriel Synnaeve
2. Xiao Liu
3. Nicolas Usunier
4. Yuxiao Dong
5. Wenhan Xiong
6. Baptiste Rozière
7. Wen Xiao
8. Thomas Scialom

In Natural Language Processing

1. Gabriel Synnaeve
2. Yossi Adi
3. Nicolas Usunier
4. Thomas Scialom
5. Louis Martin
6. Jade Copet
7. Yuxiao Dong
8. Aohan Zeng

Citations in the corpus

• Code Llama: Open Foundation Models for Code (2023, methodology): “We compare the original test set from (Guo et al., 2023) to our resampled “LCC-balanced” test set.”
• PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling (2024, background): “It includes 17 datasets covering tasks such as single-document QA [20, 6], multi-document QA [33, 16], summarization [17, 35, 10], few-shot learning [21, 13, 19], synthetic, and code generation [14, 24].”
• A Survey on Large Language Models for Code Generation (2024, unknown): “A competent data mining expert needs a foundational education in computer science or related fields along with practical experience in programming languages (PLs), such as Python and R [2].”
• Evaluating Large Language Models in Class-Level Code Generation (2024, background): “While some recent code completion efforts [31, 74] focus on handling longer code contexts with more dependencies, the generated output still remains at the function or statement level.”
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2023, methodology): “The LCC dataset is sampled from the original Long Code Completion dataset (Guo et al., 2023).”
• Gated Delta Networks: Improving Mamba2 with Delta Rule (2024, unknown): “…document summarization (GovReport (Huang et al., 2021), QMSum (Zhong et al., 2021), MultiNews (Fabbri et al., 2019)), and various specialized tasks (TRec (Li & Roth, 2002), Trivia QA (Joshi et al., 2017b), SamSum (Gliwa et al., 2019), LCC (Guo et al., 2023), and RepoBench-P (Liu et al., 2023)).”