JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation

By R. Agashe, Srini Iyer, Luke S. Zettlemoyer. 2019

92 citations recorded by the source corpus.

Interactive programming with interleaved code snippet cells and natural language markdown is recently gaining popularity in the form of Jupyter notebooks, which accelerate prototyping and collaboration. To study code generation conditioned on a long context history, we present JuICe, a corpus of 1.5 million examples with a curated test set of 3.7K instances based on online programming assignments. Compared with existing contextual code generation datasets, JuICe provides refined human-curated data, open-domain code, and an order of magnitude more training data. Using JuICe, we train models for two tasks: (1) generation of the API call sequence in a code cell, and (2) full code cell generation, both conditioned on the NL-Code history up to a particular code cell. Experiments using current baseline code generation models show that both context and distant supervision aid in generation, and that the dataset is challenging for current systems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Code Generation & Program Synthesis

Citations in the corpus

InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “We additionally removed repositories in the evaluation sets of the JuICe tasks [1], although we did not evaluate our model on these tasks in this present work.”
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis (2022, background): “, 2018) or alternating “cells” of preceding code and text blocks (Agashe et al., 2019), while the primary focus is to generate the target program itself.”
Execution-Based Evaluation for Open-Domain Code Generation (2022, methodology): “Some datasets preserve natural coverage from interactive Jupyter Notebooks (Agashe et al., 2019) or Stack-Overﬂow posts (Yin et al., 2018; Wang et al., 2022), but face challenges in enabling execution (Lai et al., 2022; Chandel et al., 2022).”
DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation (2022, background): “Notably, such a check is different from the standard surface-form metrics such as CodeBLEU (Ren et al., 2020), which requires the whole model prediction to be uniformly similar to a reference solution; instead, DS-1000 precisely targets small but important parts of surface form.”
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “Other benchmarks measure using broader APIs or general software engineering tasks, such as JuICe (Agashe et al., 2019), APIBench (Patil et al., 2023), RepoBench (Liu et al., 2023c), ODEX (Wang et al., 2022b), SWE-Bench (Jimenez et al., 2023), GoogleCodeRepo (Shrivastava et al., 2023), RepoEval…”
Execution-based Evaluation for Data Science Code Generation Models (2022, methodology): “Executing notebooks needs such data dependencies, which are not provided in JuiCe.”
PlotCoder: Hierarchical Decoding for Synthesizing Visualization Code in Programmatic Context (2021, methodology): “Unfortunately, using standard LSTM-based models and Transformer architectures (Vaswani et al., 2017) fails to solve the task, as noted in prior work (Agashe et al., 2019).”
Training and Evaluating a Jupyter Notebook Data Science Assistant (2022, methodology): “To curate the DSP dataset, we start out with a set of 448 Github repositories from the Jupyter Interactive Computing (JuICe) (Agashe, Iyer, and Zettlemoyer 2019) development dataset, which contains of 33K nbgrader notebooks.”
Incorporating External Knowledge through Pre-training for Natural Language to Code Generation (2020, background): “However, recently there has been a move towards applying semantic parsing to automatically generating source code in general-purpose programming languages (Yin et al., 2018; Yao et al., 2018; Lin et al., 2018; Agashe et al., 2019; Yao et al., 2019).”
L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models (2023, background): “, 2020), coding assistant (Agashe et al., 2019b; Lai et al., 2022), language interfaces to databases (Pasupat and Liang, 2015; Yu et al.”
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2023, methodology): “By following a similar approach, JuICe (Agashe et al., 2019) collects publicly available Jupyter notebooks from GitHub, and CoNaLa (Yin et al., 2018) collects Python and Java codes with NL comments from StackOverﬂow posts.”
Natural Language to Code Generation in Interactive Data Science Notebooks (2022, methodology): “1, data scientists spend signiﬁcant amount of time on data wrangling and exploratory data analysis (EDA) (Agashe et al., 2019; Wang cutable programs (Chen et al., 2021a; Austin et al., 2021; Chowdhery et al., 2022; Nijkamp et al., 2022; Fried et al., 2022, inter alia ), with promising applications…”
In-IDE Code Generation from Natural Language: Promise and Challenges (2021, methodology): “Specifically, many code generation methods [1, 47, 115, 120] aim to directly compare generated code snippets against ground truth snippets, using token sequence comparison metrics borrowed from machine translation tasks, such as BLEU score [86].”
Out of the BLEU: how should we assess quality of the Code Generation models? (2022, methodology): “We focus on the datasets containing general Python code, leaving the non-Python datasets such as Spider (containing SQL) [7] and JuICe (containing Jupyter Notebooks) [37] out of the scope.”