Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language

By Paul Denny, Viraj Kumar, Nasser Giacaman. 2022

328 citations recorded by the source corpus.

GitHub Copilot is an artificial intelligence tool for automatically generating source code from natural language problem descriptions. Since June 2022, Copilot has officially been available for free to all students as a plug-in to development environments like Visual Studio Code. Prior work exploring OpenAI Codex, the underlying model that powers Copilot, has shown it performs well on typical CS1 problems thus raising concerns about its potential impact on how introductory programming courses are taught. However, little is known about the types of problems for which Copilot does not perform well, or about the natural language interactions that a student might have with Copilot when resolving errors. We explore these questions by evaluating the performance of Copilot on a publicly available dataset of 166 programming problems. We find that it successfully solves around half of these problems on its very first attempt, and that it solves 60% of the remaining problems using only natural language changes to the problem description. We argue that this type of prompt engineering, which we believe will become a standard interaction between human and Copilot when it initially fails, is a potentially useful learning activity that promotes computational thinking skills, and is likely to change the nature of code writing skill development.

Subfields: Computer Science Applications, Information Systems, Software, Prompting & Reasoning, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Information Systems

In Software

In Prompting & Reasoning

In Code Generation & Program Synthesis

Citations in the corpus

Comparing Code Explanations Created by Students and Large Language Models (2023, background): “Similar work involving a public dataset of programming problems found that Codex produced correct solutions on its first attempt approximately half of the time, increasing to 80% when repeated attempts and minor adjustments to the input prompt were allowed [8].”
Parsel: A Unified Natural Language Framework for Algorithmic Reasoning (2022, background): “Those which cannot be solved currently will be increasingly solved (Denny et al., 2022).”
Computing Education in the Era of Generative AI (2023, background): “Thus, AI-generated solutions can be a time-efficient option [11].”
Large Language Models for Software Engineering: Survey and Open Problems (2023, background): “Denny et al. [60] argued that prompt engineering is a useful learning activity that fosters software engineering students’ computational thinking.”