On the difficulty of benchmarking inductive program synthesis methods
By Edward R. Pantridge, Thomas Helmuth, Nicholas Freitag McPhee, Lee Spector. 2017
20 citations recorded by the source corpus.
Subfields: Artificial Intelligence, Computer Science Applications, Hardware and Architecture, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Luke S. Zettlemoyer
- Quoc V. Le
- Mohit Bansal
- Pieter Abbeel
- Percy Liang
- Jianfeng Gao
- Yejin Choi
In Computer Science Applications
In Hardware and Architecture
In Evaluation & Benchmarks
- Michael O'Neill
- Christopher Hesse
- Harrison Edwards
- Lee Spector
- Ilya Sutskever
- Wojciech Zaremba
- Jan Leike
- Joshua Achiam
Citations in the corpus
- Evaluating Large Language Models Trained on Code (2021, methodology): “While evaluating code generation is well-studied (Xu et al., 2021; Helmuth & Spector, 2015; Pantridge et al., 2017), many existing metrics measure performance in tightly specified, constrained problem instances (e.”
- A Hazard Analysis Framework for Code Synthesis Large Language Models (2022, neutral): “Evaluating the capabilities of code synthesis and generation is not a novel problem and has been explored in both the Machine Learning (ML) [35] and Synthesis [20, 21, 30] communities.”
- Automatic programming: The open issue? (2019, methodology): “study comparing multiple approaches to program synthesis [51], the General Program Synthesis Benchmark Suite [19] and a set of Basic Execution Model problems were employed.”