Learning and Evaluating Contextual Embedding of Source Code

By Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, Kensen Shi. 2019

452 citations recorded by the source corpus.

Subfields: Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Program Synthesis with Large Language Models (2021, methodology): “Several pre-trained models for code have shown to be effective for transfer learning across software engineering tasks, including CuBERT [Kanade et al., 2020], CodeBERT [Feng et al.”
InCoder: A Generative Model for Code Infilling and Synthesis (2022, neutral): “, decoder-only language models [10, 18, 35, 68, 46], encoder-only masked language models [24, 37], and encoder-decoder models [4, 44, 54, 64].”
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis (2022, methodology): “Prior works, such as CuBERT (Kanade et al., 2020), CodeBERT (Feng et al.”
StarCoder: may the source be with you! (2023, methodology): “…developed in NLP were also applied successfully to code, including encoder-only models for producing code representations (Feng et al., 2020; Kanade et al., 2020) and encoder-decoder models for translation, editing, summarization, and language-to-code tasks (Wang et al., 2021; Ahmad et al.,…”
SantaCoder: don't reach for the stars! (2023, neutral): “…Christopoulou et al., 2022; Izadi et al., 2022; Xu et al., 2022b; Athiwaratkun et al., 2022), while other studies have investigated encoder (Feng et al., 2020a; Kanade et al., 2020) and encoder-decoder architectures (Li et al., 2022; Ahmad et al., 2021; Wang et al., 2021; Roziere et al., 2021).”
The Stack: 3 TB of permissively licensed source code (2022, background): “Researchers have also investigated encoder masked language models (Feng et al., 2020; Kanade et al., 2020) and encoder-decoder architectures with various training objectives (Li et al., 2022; Ahmad et al., 2021; Wang et al., 2021; Roziere et al., 2021).”
A systematic evaluation of large language models of code (2022, background): “, 2021) and CuBERT (345M) (Kanade et al., 2020) are examples of such models in code.”
CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2021, background): “Pre-trained language models such as BERT (De-vlin et al., 2019), GPT (Radford et al., 2019), and T5 (Raffel et al., 2020) have greatly boosted performance in a wide spectrum of natural language processing (NLP) tasks.”
DOBF: A Deobfuscation Pre-Training Objective for Programming Languages (2021, methodology): “Kanade et al. (2020) applied MLM and the next sentence prediction objectives to pre-train models on Python code.”
A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, methodology): “CuBERT [192] shares similar model architecture with CodeBERT meanwhile it is not required to perform sentence separation between the natural-language description of a function and its body for sentence-pair representation.”
TU_DBS in the ARQMath Lab 2021, CLEF (2021, methodology): “As a consequence, various models for different domains have been developed, such as BioBERT [12], ClinicalBERT [13, 14] or SciBERT [15] for scientific domains or CuBERT [16] and CodeBERT [17].”
Large Language Models for Software Engineering: A Systematic Literature Review (2023, background): “We also face issues like the need for extensive labeled data [347], significant computational resources [118, 248], and avoiding potential ethical problems [5, 40, 325].”
Leveraging Automated Unit Tests for Unsupervised Code Translation (2021, methodology): “Unsupervised pre-training methods for code based on BERT (Kanade et al., 2020; Feng et al., 2020), BART (Ahmad et al., 2021) or other objectives tailored to source code (Guo et al., 2020; Roziere et al., 2021) have shown strong results on benchmarks such as CodeXGLUE (Lu et al., 2021).”
Repository-Level Prompt Generation for Large Language Models of Code (2022, methodology): “Encoder-only models use a masked language modeling ob-jective (Feng et al., 2020; Guo et al., 2020; Kanade et al., 2020).”
Combiner: Full Attention Transformer with Sparse Computation Cost (2021, background): “It has also achieved excellent results in other domains like image recognition [9], code understanding [10], speech recognition [11], protein [12], music [13] and image [14] generative modeling.”
Learning to Predict User-Defined Types (2023, methodology): “This pre-training approach processes large swaths of raw source code available on GitHub [20], [21], [22], [23], [24] to learn representations (neural encodings) which capture common usage patterns.”
Learning type annotation: is big data enough? (2021, background): “Feng et al. [8] used pre-training to improve performance on code-natural language bi-modal datasets ( e.g. code with comments) and Kanade et al. [12] used pre-training to help with retrieval-like tasks.”
Multilingual training for Software Engineering (2021, background): “This approach is known toachieve state-of-the-art performance in both natural language processing, and software-related tasks [3, 4, 11, 18, 22, 32, 33, 35, 50, 75].”