A large-scale benchmark for few-shot program induction and synthesis

By Ferran Alet, Javier Lopez-Contreras, James Koppel, Maxwell Nye, Armando Solar-Lezama, Tomas Lozano-Perez, Leslie Pack Kaelbling, Josh Tenenbaum. 2021

20 citations recorded by the source corpus.

Subfields: Evaluation & Benchmarks, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

1. Fei Xia
2. Jie Tan
3. Dorsa Sadigh
4. Shao-Hua Sun
5. Peng Xu
6. Adam Tauman Kalai
7. Augustus Odena
8. Nicolas Manfred Otto Heess

In Code Generation & Program Synthesis

1. Charles Sutton
2. Augustus Odena
3. Adam Tauman Kalai
4. Quoc V. Le
5. Maxwell Nye
6. Miltiadis Allamanis
7. Premkumar T. Devanbu
8. Michael Pradel

Citations in the corpus

• Program Synthesis with Large Language Models (2021, methodology): “Such datasets include the Natural Program Synthesis (NAPS) dataset [Zavershynskyi et al., 2018], the Search-based Pseudocode to Code (SPoC) dataset [Kulal et al., 2019], the APPS dataset [Hendrycks et al., 2021], the PROGRES dataset [Alet et al., 2021], and the CodeNet dataset [Puri et al., 2021].”
• Learning to Synthesize Programs as Interpretable and Generalizable Policies (2021, background): “[52] Ferran Alet, Javier Lopez-Contreras, James Koppel, Maxwell Nye, Armando Solar-Lezama, Tomas Lozano-Perez, Leslie Kaelbling, and Joshua Tenenbaum.”
• Language Models Can Teach Themselves to Program Better (2022, methodology): “As a baseline, we compare to GPT-Neo with no fine-tuning, which solves few of P3’s test puzzles.”
• Language to Rewards for Robotic Skill Synthesis (2023, background): “The capability of those models range from solving coding competition questions [38] and benchmarks [39], to drawing simple figures [40], generating policies that solve 2D tasks [41], and complex instruction following tasks [3].”