Training and Evaluating a Jupyter Notebook Data Science Assistant

By Shubham Chandel, Colin B. Clement, Guillermo Serrato, Neel Sundaresan. 2022

56 citations recorded by the source corpus.

We study the feasibility of a Data Science assistant powered by a sequence-to-sequence transformer by training a new model JuPyT5 on all publicly available Jupyter Notebook GitHub repositories and developing a new metric: Data Science Problems (DSP). DSP is a collection of 1119 problems curated from 306 pedagogical notebooks with 92 dataset dependencies, natural language and Markdown problem descriptions, and assert-based unit tests. These notebooks were designed to test university students' mastery of various Python implementations of Math and Data Science, and we now leverage them to study the ability of JuPyT5 to understand and pass the tests. We analyze the content of DSP, validate its quality, and we find that given 100 sampling attempts JuPyT5 is able to solve 77.5\% of the DSP problems. We further present various ablation and statistical analyses and compare DSP to other recent natural language to code benchmarks.

Subfields: Artificial Intelligence, Information Systems, Statistics and Probability, Code Generation & Program Synthesis, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Statistics and Probability

In Code Generation & Program Synthesis

In Human-Computer Interaction

Jianfeng Gao

Citations in the corpus

Execution-Based Evaluation for Open-Domain Code Generation (2022, background): “Some datasets preserve natural coverage from interactive Jupyter Notebooks (Agashe et al., 2019) or Stack-Overﬂow posts (Yin et al., 2018; Wang et al., 2022), but face challenges in enabling execution (Lai et al., 2022; Chandel et al., 2022).”
DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation (2022, background): “…science coding problems usually have diverse contexts including their incorrect code, error messages, and input-output examples, which cannot be found in most prior data science relevant code generation benchmarks (Yin et al., 2018; Hendrycks et al., 2021; Chandel et al., 2022; Chen et al., 2021a).”
A Survey on Large Language Models for Code Generation (2024, unknown): “The rapid development and huge potential capabilities of LLMs also raise several significant limitations and challenges.”
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2023, background): “In addition, most of the related works (Hendrycks et al., 2021; Lai et al., 2023; Huang et al., 2022; Nijkamp et al., 2023; Chandel et al., 2022) focused on code generation tasks with target as Python when evaluating model performance, mainly because setting up a Python execution environment is…”
Execution-based Evaluation for Data Science Code Generation Models (2022, methodology): “(2) Surprisingly, zero-shot Codex achieves compatible results with ﬁnetuned JuPyT5 in OutputEM, but it performs badly with surface-form metrics.”
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2023, methodology): “This execution-based paradigm has been widely applied to evaluate benchmarks in Data Science domain like DSP (Chandel et al., 2022), DS-1000 (Lai et al., 2022) and Exe-DS (Huang et al., 2022) as well as general code generation benchmarks in single-language settings such as HumanEval (Chen et al., 20”
Natural Language to Code Generation in Interactive Data Science Notebooks (2022, methodology): “Other datasets assume that the developer provides extra information, such as unit tests or input/output examples (Chandel et al., 2022; Jain et al., 2022), but such systems pose an extra burden to users who might not normally write such tests or examples during their workﬂow (Pimentel et al., 2019).”