Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming

By Majeed Kazemitabaar, Justin Chow, Carlsen Ma, B. Ericson, David Weintrop, Tovi Grossman. International Conference on Human Factors in Computing Systems, 2023

390 citations recorded by the source corpus.

AI code generators like OpenAI Codex have the potential to assist novice programmers by generating code from natural language descriptions, however, over-reliance might negatively impact learning and retention. To explore the implications that AI code generators have on introductory programming, we conducted a controlled experiment with 69 novices (ages 10-17). Learners worked on 45 Python code-authoring tasks, for which half of the learners had access to Codex, each followed by a code-modification task. Our results show that using Codex significantly increased code-authoring performance (1.15x increased completion rate and 1.8x higher scores) while not decreasing performance on manual code-modification tasks. Additionally, learners with access to Codex during the training phase performed slightly better on the evaluation post-tests conducted one week later, although this difference did not reach statistical significance. Of interest, learners with higher Scratch pre-test scores performed significantly better on retention post-tests, if they had prior access to Codex.

Subfields: Computer Science Applications, Information Systems

View Paper: https://www.semanticscholar.org/paper/f03f39f735186c4359b719724be6e1c2eb912fef

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Information Systems

Citations in the corpus

CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs (2024, methodology): “Kazemitabaar et al. developed Coding Steps to explore the use of LLM-based code generators for supporting learners in introductory programming [27].”
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023, background): “studied how novice programmer productivity and learning is affected by the use of AI code generators [101].”
Computing Education in the Era of Generative AI (2023, background): “Depending on the prompt, the resulting programs used varied programmatic structures, while ultimately favoring expected methods for each problem variation (i.e., for-loops for processing lists, and while-loops for processing standard input).”