Natural Language to Code Generation in Interactive Data Science Notebooks

By Pengcheng Yin, Wen-Ding Li, Kefan Xiao, A. Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski and 2 more. 2022

122 citations recorded by the source corpus.

Computational notebooks, such as Jupyter notebooks, are interactive computing environments that are ubiquitous among data scientists to perform data wrangling and analytic tasks. To measure the performance of AI pair programmers that automatically synthesize programs for those tasks given natural language (NL) intents from users, we build ARCADE, a benchmark of 1078 code generation problems using the pandas data analysis framework in data science notebooks. ARCADE features multiple rounds of NL-to-code problems from the same notebook. It requires a model to understand rich multi-modal contexts, such as existing notebook cells and their execution states as well as previous turns of interaction. To establish a strong baseline on this challenging task, we develop PaChiNCo, a 62B code language model (LM) for Python computational notebooks, which significantly outperforms public code LMs. Finally, we explore few-shot prompting strategies to elicit better code with step-by-step decomposition and NL explanation, showing the potential to improve the diversity and explainability of model predictions. Arcade is publicly available at https://github.com/google-research/arcade-nl2code/.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

PaLM 2 Technical Report (2023, methodology): “As in (Chen et al., 2021; Austin et al., 2021; Yin et al., 2022), we benchmark models in a pass@1 and pass@k setting.”
Mathematical discoveries from program search with large language models (2023, methodology): “There are many use cases being explored, such as automatically editing code to improve performance 13 , automatically debugging code 9,10 , generating code from natural language descriptions 69–71 and doing so to solve problems in code competitions 11,12 .”
Agentless: Demystifying LLM-based Software Engineering Agents (2024, methodology): “This is done by providing the previously described repository structure and asking the LLM to produce a list of irrelevant folders that do not need to be further inspected or modified to resolve the issue.”
Parsel: A Unified Natural Language Framework for Algorithmic Reasoning (2022, background): “One alternative that has been explored is conversational code generation (Nijkamp et al., 2022; Yin et al., 2022).”
SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023, background): “Simultaneously, separate works have sought to introduce new coding paradigms (Yin et al., 2022; Yang et al., 2023) or design library-specific problems (Lai et al.”
CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis (2025, unknown): “…al., 2023; Jain et al., 2024a; Banerjee et al., 2025; Ni et al., 2024b; Liu et al., 2023d; Du et al., 2023; Suresh et al., 2025b; Zhuo et al., 2024; Yin et al., 2022; Lai et al., 2023; Patil et al., 2025; Ugare et al., 2024; 2025; Wei et al., 2025d; Wu et al., 2025), evaluate code generation from…”