COSET: A Benchmark for Evaluating Neural Program Embeddings
By Ke Wang, Mihai Christodorescu. 2019
41 citations recorded by the source corpus.
Neural program embedding can be helpful in analyzing large software, a task that is challenging for traditional logic-based program analyses due to their limited scalability. A key focus of recent machine-learning advances in this area is on modeling program semantics instead of just syntax. Unfortunately evaluating such advances is not obvious, as program semantics does not lend itself to straightforward metrics. In this paper, we introduce a benchmarking framework called COSET for standardizing the evaluation of neural program embeddings. COSET consists of a diverse dataset of programs in source-code format, labeled by human experts according to a number of program properties of interest. A point of novelty is a suite of program transformations included in COSET. These transformations when applied to the base dataset can simulate natural changes to program code due to optimization and refactoring and can serve as a "debugging" tool for classification mistakes. We conducted a pilot study on four prominent models: TreeLSTM, gated graph neural network (GGNN), AST-Path neural network (APNN), and DYPRO. We found that COSET is useful in identifying the strengths and limitations of each model and in pinpointing specific syntactic and semantic characteristics of programs that pose challenges.
Subfields: Artificial Intelligence, Information Systems, Software, Evaluation & Benchmarks, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Kyunghyun Cho
- Quoc V. Le
- Pieter Abbeel
- Percy Liang
In Information Systems
- Joseph E. Gonzalez
- Raymond Li
- Goutham Ramakrishnan
- Harm de Vries
- Ajay Jain
- Pieter Abbeel
- Aws Albarghouthi
- Tianjun Zhang
In Software
- Goutham Ramakrishnan
- Ajay Jain
- Pieter Abbeel
- Aws Albarghouthi
- Tianjun Zhang
- Paras Jain
- Ion Stoica
- Joseph E. Gonzalez
In Evaluation & Benchmarks
- Harm de Vries
- Ajay Jain
- Pieter Abbeel
- Tianjun Zhang
- Paras Jain
- Yacine Jernite
- Ion Stoica
- Dzmitry Bahdanau
In Code Generation & Program Synthesis
Citations in the corpus
- The Stack: 3 TB of permissively licensed source code (2022, background): “CoSet (Wang & Christodorescu, 2019) introduced a dataset of 84,165 programs for semantic evaluation of neural program embeddings.”
- Backdoors in Neural Models of Source Code (2020, background): “Several works have studied test-time adversarial attacks on models of code [28, 29, 30, 31].”
- Contrastive Code Representation Learning (2020, result): “5955 we show that RoBERTa’s representations of code are sensitive to code edits in agreement with prior work (Wang and Christodorescu, 2019; Wang and Su, 2019; Rabin and Alipour, 2020).”