Can Students Review Their Peers?: Comparison of Peer and Instructor Reviews

By Nea Pirttinen, Juho Leinonen. 2022

16 citations recorded by the source corpus.

Having students peer review each other's exercises is a common task in modern computing classrooms. In large classes, peer review might even partly replace traditional instructor-led review -- and prior work has found some indications that the quality of peer reviews can be close to that of instructor reviews. In this work, we explore the difference between instructor and peer reviews of student-created programming exercises. One task in an introductory programming course was to have students design their own programming exercises -- including an exercise description, model solution, and test cases -- which were then reviewed by peers. After the course, we had two instructors review the same student-created exercises. We compare the scores given by the instructors and the students to analyze potential differences. Our results suggest that agreement between instructors and students as measured by inter-rater reliability is low, although differences between instructor and student review score distributions are not statistically significant. Additionally, instructors have more fluctuation in their reviews compared to students. Due to the rising popularity of peer reviews, more research is needed to examine to what extent they could complement traditional instructor-led review of exercises.

Subfields: Computer Science Applications, Education, Human-Computer Interaction, Social Sciences & Humanities

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

1. Paul Denny
2. Juho Leinonen
3. Sami Sarsa
4. Arto Hellas
5. Stephen MacNeil
6. Seth Bernstein
7. Andrew Tran
8. Joanne Kim

In Education

1. Paul Denny
2. Sami Sarsa
3. Arto Hellas
4. Juho Leinonen

In Human-Computer Interaction

1. Sami Sarsa
2. Juho Leinonen
3. Arto Hellas
4. Paul Denny

Citations in the corpus

• Comparing Code Explanations Created by Students and Large Language Models (2023, neutral): “For example, prior work has found differences in how students and instructors rate learnersourced programming exercises [35].”
• Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models (2022, background): “This would address one of the major challenges related to the use of learnersourcing which is that students tend to be much more inclined to use and evaluate resources created by others than they are to create resources themselves [66, 78].”
• Robosourcing Educational Resources - Leveraging Large Language Models for Learnersourcing (2022, background): “Indeed, prior research has shown that students tend to be much more inclined to use and evaluate resources created by others than they are to create resources themselves [9, 10, 11].”