Jigsaw: Large Language Models meet Program Synthesis
By Naman Jain, Skanda Vaidyanath, Arun Shankar Iyer, Nagarajan Natarajan, Suresh Parthasarathy, Sriram K. Rajamani, Rahul Sharma. 2021
257 citations recorded by the source corpus.
Large pre-trained language models such as GPT-3 [10], Codex [11], and Coogle's language model [7] are now capable of generating code from natural language specifications of programmer intent. We view these developments with a mixture of optimism and caution. On the optimistic side, such large language models have the potential to improve productivity by providing an automated AI pair programmer for every programmer in the world. On the cautionary side, since these large language models do not understand program semantics, they offer no guarantees about quality of the suggested code. In this paper, we present an approach to augment these large language models with post-processing steps based on program analysis and synthesis techniques, that understand the syntax and semantics of programs. Further, we show that such techniques can make use of user feedback and improve with usage. We present our experiences from building and evaluating such a tool Jigsaw, targeted at synthesizing code for using Python Pandas API using multi-modal inputs. Our experience suggests that as these large language models evolve for synthesizing code from intent, Jigsaw has an important role to play in improving the accuracy of the systems.
Subfields: Artificial Intelligence, Information Systems, Software, Large Language Models, LLM Agents & Tool Use
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Graham Neubig
- Uri Alon
- Luke S. Zettlemoyer
- Mohit Bansal
- Percy Liang
- Jianfeng Gao
- Charles Sutton
- Yoshua Bengio
In Information Systems
In Software
- Koushik Sen
- Mark Harman
- Oleksandr Polozov
- Shin Yoo
- Armando Solar-Lezama
- Charles Sutton
- Kensen Shi
- Uri Alon
In Large Language Models
- Graham Neubig
- Armando Solar-Lezama
- Mark Harman
- Naman Jain
- Tianjun Zhang
- Aman Madaan
- Ion Stoica
- Kensen Shi
Citations in the corpus
- Large Language Models Are Human-Level Prompt Engineers (2022, background): “Modern program synthesis admits a wide variety of specifications, including input-output examples (Ellis et al., 2021; Wong et al., 2021) and natural language (Jain et al., 2022).”
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “Benchmarks like DS-1000 (Lai et al., 2023), ARCADE (Yin et al., 2022), NumpyEval (Zhang et al., 2023b), and PandasEval (Jain et al., 2022) focus on data science APIs.”
- Large Language Models for Software Engineering: Survey and Open Problems (2023, methodology): “Jain et al. [71] proposed Jigsaw, which post-processes the generated code based on program analysis and synthesis techniques.”
- GPT (Generative Pre-Trained Transformer)— A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions (2023, background): “GPT is capable of interpreting the queries, providing important details like order status, tracking information, and expected delivery time, as well as suggesting corrections for frequent problems [134].”
- Natural Language to Code Generation in Interactive Data Science Notebooks (2022, methodology): “, 2021c), followed by recent efforts using LLMs with additional NL specifications (Jain et al., 2021; Bavishi, 2022).”
- Gorilla: Large Language Model Connected with Massive APIs (2023, unknown): “Harnessing LLMs for program synthesis has historically been a challenging task [22, 7, 45, 16, 13, 29].”
- Learning Performance-Improving Code Edits (2023, background): “Recent work leverages machine learning to assist program synthesis (Balog et al., 2016; Bunel et al., 2018; Devlin et al., 2017a;b; Kalyan et al., 2018; Parisotto et al., 2016; Odena & Sutton, 2020; Nye et al., 2019; Meng et al., 2011; Jain et al., 2022; Austin et al., 2021; Xiong & Wang, 2022; Camb”
- Large Language Model guided Protocol Fuzzing (2024, unknown): “LLMs have also been proven effective in translating natural language specifications and instructions into executable code [20], [24], [14].”
- Across Programming Language Silos: A Study on Cross-Lingual Retrieval-augmented Code Generation (2025, unknown): “With the emergence and continuous development of large language models (LLMs), significant progress has been made in natural language (NL) to code generation task [1]–[4].”