Crowdsourcing programming assignments with CrowdSorcerer

By Nea Pirttinen, Vilma Kangas, Irene Nikkarinen, Henrik Nygren, Juho Leinonen, Arto Hellas. 2018

28 citations recorded by the source corpus.

Small automatically assessed programming assignments are an often used resource for learning programming. Creating sufficiently large amounts of such assignments is, however, time consuming. As a consequence, offering large quantities of practice assignments to students is not always possible. CrowdSorcerer is an embeddable open-source system that students and teachers alike can use for creating and evaluating small automatically assessed programming assignments. While creating programming assignments, the students also write simple input-output -tests, and are gently introduced to the basics of testing. Students can also evaluate the assignments of others and provide feedback on them, which exposes them to code written by others early in their education. In this article we both describe the CrowdSorcerer system and our experiences in using the system in a large undergraduate programming course. Moreover, we discuss the motivation for crowdsourcing course assignments and present some usage statistics.

Subfields: Computer Science Applications, Software Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

1. Juho Leinonen
2. Paul Denny
3. Sami Sarsa
4. Arto Hellas
5. Stephen MacNeil
6. Seth Bernstein
7. Andrew Tran
8. Joanne Kim

In Software Engineering

1. Juho Leinonen
2. Nea Pirttinen
3. Arto Hellas
4. Stephen MacNeil
5. Seth Bernstein
6. Andrew Tran

Citations in the corpus

• Comparing Code Explanations Created by Students and Large Language Models (2023, background): “While learnersourcing techniques could be used to generate code explanations efficiently by directly involving students in their creation [23, 34], there are known issues relating to quality when learning content is sourced from students [1, 9].”
• Crowdsourcing Content Creation for SQL Practice (2020, methodology): “Another way to alleviate quality issues is peer reviewing, as with the multiple-choice questions in PeerWise [12] and programming assignments in CrowdSorcerer [31].”
• Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book (2022, methodology): “Learnersourcing has been used successfully in computing education to create multiple choice questions [8], programming exercises [9, 33], and SQL exercises [21].”
• Can Students Review Their Peers?: Comparison of Peer and Instructor Reviews (2022, methodology): “In CrowdSorcerer, students can create programming exercises according to instructions given by the instructor of the course.”
• Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models (2022, background): “One instructional approach that has become increasingly common in computing education is learnersourcing [41] where students participate in the creation and evaluation of course materials such as questions and exercises (see e.g. [14, 16, 47, 65]).”