NAPS: Natural Program Synthesis Dataset

By Maksym Zavershynskyi, Alexander Skidanov, Illia Polosukhin. 2018

37 citations recorded by the source corpus.

We present a program synthesis-oriented dataset consisting of human written problem statements and solutions for these problems. The problem statements were collected via crowdsourcing and the program solutions were extracted from human-written solutions in programming competitions, accompanied by input/output examples. We propose using this dataset for the program synthesis tasks aimed for working with real user-generated data. As a baseline we present few models, with the best model achieving 8.8% accuracy, showcasing both the complexity of the dataset and large room for future research.

Subfields: Artificial Intelligence, Information Systems, Software, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Software

In Evaluation & Benchmarks

Citations in the corpus

Program Synthesis with Large Language Models (2021, methodology): “Such datasets include the Natural Program Synthesis (NAPS) dataset [Zavershynskyi et al., 2018], the Search-based Pseudocode to Code (SPoC) dataset [Kulal et al., 2019], the APPS dataset [Hendrycks et al., 2021], the PROGRES dataset [Alet et al., 2021], and the CodeNet dataset [Puri et al., 2021].”
SPoC: Search-based Pseudocode to Code (2019, background): “And in contrast to the closely related NAPS dataset [41], which also contains test cases but only 6% human-authored pseudocode, all programs in SPoC are associated with human-authored pseudocode of a consistent annotation granularity.”
Measuring Coding Challenge Competence With APPS (2021, methodology): “Zavershynskyi et al. (2018) introduce the NAPS dataset for converting pseudocode to code, obtained by crowdsourcing low-level descriptions of programming exercises, and apply machine translation techniques to the problem.”
Execution-based Evaluation for Data Science Code Generation Models (2022, unknown): “, 2021), and input-output examples (Polosukhin and Skidanov, 2018; Zavershynskyi et al., 2018).”
PlotCoder: Hierarchical Decoding for Synthesizing Visualization Code in Programmatic Context (2021, methodology): “…information such as database schemas (Zhong et al., 2017; Yu et al., 2018, 2019b,a) or input-output examples (Polo-sukhin and Skidanov, 2018; Zavershynskyi et al., 2018), synthesizing general-purpose code from nat-ural language remains highly difﬁcult due to the ambiguity in the natural…”
Semantic Scaffolds for Pseudocode-to-Code Generation (2020, methodology): “…behavior in a game engine, the CONCODE dataset (Iyer et al., 2018) consisting of Java documentation strings and method bodies, and the NAPS and SPoC datasets (Zavershynskyi et al., 2018; Kulal et al., 2019) consisting of pseudocode annotations and source code for programming competition problems.”
A large-scale benchmark for few-shot program induction and synthesis (2021, background): “Zavershynskyi et al. (2018) is probably closest to our benchmark, combining a mixture of crowd-sourced descriptions of subprograms with input-output examples for the entire programs (not subprograms).”
Programming Puzzles (2021, background): “English descriptions, often mixed with examples, are becoming an increasingly popular problem representation as LMs improve [28, 33, 60].”