General Program Synthesis Benchmark Suite

By Thomas Helmuth, Lee Spector. 2015

174 citations recorded by the source corpus.

Recent interest in the development and use of non-trivial benchmark problems for genetic programming research has highlighted the scarcity of general program synthesis (also called "traditional programming") benchmark problems. We present a suite of 29 general program synthesis benchmark problems systematically selected from sources of introductory computer science programming problems. This suite is suitable for experiments with any program synthesis system driven by input/output examples. We present results from illustrative experiments using our reference implementation of the problems in the PushGP genetic programming system. The results show that the problems in the suite vary in difficulty and can be useful for assessing the capabilities of a program synthesis system.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

In Code Generation & Program Synthesis

Citations in the corpus

Evaluating Large Language Models Trained on Code (2021, methodology): “While evaluating code generation is well-studied (Xu et al., 2021; Helmuth & Spector, 2015; Pantridge et al., 2017), many existing metrics measure performance in tightly speciﬁed, constrained problem instances (e.g., string manipulation in FlashFill (Gul-wani, 2011)).”
A Hazard Analysis Framework for Code Synthesis Large Language Models (2022, neutral): “Evaluating the capabilities of code synthesis and generation is not a novel problem and has been explored in both the Machine Learning (ML) [35] and Synthesis [20, 21, 30] communities.”
Program synthesis using uniform mutation by addition and deletion (2018, methodology): “These experiments use problems from a recent program synthesis benchmark suite, composed of 29 problems from introductory computer science textbooks [6].”
On the difficulty of benchmarking inductive program synthesis methods (2017, methodology): “When the original benchmark paper [8] was published, PushGP had not been able to find solutions to the Checksum problem.”
On End-to-End Program Generation from User Intention by Deep Neural Networks (2015, background): “Program generation by genetic programming [20, 7] can automatically search the space of candidate programs (ineﬃciently), but carefully chosen mutation or crossover operations should also be provided.”
Automatic programming: The open issue? (2019, methodology): “In the recent Pantridge et al. study comparing multiple approaches to program synthesis [51], the General Program Synthesis Benchmark Suite [19] and a set of Basic Execution Model problems were employed.”