Evaluating the evaluations of code recommender systems: A reality check

By Sebastian Proksch, Sven Amann, Sarah Nadi, Mira Mezini. 2016

27 citations recorded by the source corpus.

Subfields: Computer Networks and Communications, Information Systems, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

1. Charles Sutton
2. Earl T. Barr
3. Vincent J. Hellendoorn
4. Alberto Bacchelli
5. Harald C. Gall
6. Premkumar T. Devanbu
7. Sebastian Proksch
8. Miltiadis Allamanis

In Information Systems

1. Charles Sutton
2. Miltiadis Allamanis
3. Earl T. Barr
4. Premkumar T. Devanbu
5. Vincent J. Hellendoorn
6. Alberto Bacchelli
7. Rafael-Michael Karampatsis
8. Michael Pradel

In Evaluation & Benchmarks

1. Miltiadis Allamanis
2. Charles Sutton
3. Earl T. Barr
4. Premkumar T. Devanbu
5. Alberto Bacchelli
6. Sebastian Proksch
7. Vincent J. Hellendoorn
8. Harald C. Gall

Citations in the corpus

• A Survey of Machine Learning for Big Code and Naturalness (2017, background): “However, developers rarely write code in such a simple and consistent way [158].”
• WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “These models are typically trained on data collected from existing codebases but then deployed more generally across other codebases, which may have different distributions of API usages (Nita & Notkin, 2010; Proksch et al., 2016; Allamanis & Brockschmidt, 2017).”
• When Code Completion Fails: A Case Study on Real-World Completions (2019, methodology): “Synthetic code completion benchmarks are typically created by taking a complete program and removing a random token, such as an identifier or method invocation [4], [9]–[12], [14].”