Tuned Models of Peer Assessment in MOOCs

By C. Piech, Jonathan Huang, Zhenghao Chen, Chuong B. Do, Andrew Y. Ng, Daphne Koller. Educational Data Mining, 2013

416 citations recorded by the source corpus.

In massive open online courses (MOOCs), peer grading serves as a critical tool for scaling the grading of complex, open-ended assignments to courses with tens or hundreds of thousands of students. But despite promising initial trials, it does not always deliver accurate results compared to human experts. In this paper, we develop algorithms for estimating and correcting for grader biases and reliabilities, showing significant improvement in peer grading accuracy on real data with 63,199 peer grades from Coursera's HCI course offerings --- the largest peer grading networks analysed to date. We relate grader biases and reliabilities to other student factors such as student engagement, performance as well as commenting style. We also show that our model can lead to more intelligent assignment of graders to gradees.

Subfields: Artificial Intelligence, Computer Science Applications, Information Systems

View Paper: https://www.semanticscholar.org/paper/4b94c49f86d27fd0104898a446280d4cf880b16f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Information Systems

Citations in the corpus

On the Theory and Practice of Privacy-Preserving Bayesian Data Analysis (2016, methodology): “Probabilistic models trained via Bayesian inference are widely and successfully used in application domains where privacy is invaluable, from text analysis (Blei et al., 2003; Goldwater and Grifﬁths, 2007), to personalization (Salakhutdinov and Mnih, 2008), to medical informatics (Husmeier et al., 2”
Avoiding Imposters and Delinquents: Adversarial Crowdsourcing and Peer Prediction (2016, background): “…arise in crowdsourcing (Vuurens et al., 2011), collaborative knowledge generation (Priedhorsky et al., 2007), peer grading in online classrooms (Piech et al., 2013; Kulkarni et al., 2015), aggregation of customer reviews (Harmon, 2004), and the generation/curation of large datasets (Deng et…”
Understanding Student Motivation, Behaviors and Perceptions in MOOCs (2015, background): “Other researchers have investigated the challenges of peer assessment [15, 27, 33].”
Efficient Mechanisms for Peer Grading and Dueling Bandits (2018, methodology): “…in the course (Aziz et al., 2016; Caragiannis et al., 2015, 2016a; de Alfaro and Shavlovsky, 2014; Kulkarni et al., 2013; Kurokawa et al., 2015; Piech et al., 2013; Raman and Joachims, 2014; Shah et al., 2013; Walsh, 2014), and there have been several tools developed, such as crowdgrader.org…”
Sentiment Analysis in MOOC Discussion Forums: What does it tell us? (2014, background): “But it does not always deliver accurate results compared to human experts [22].”
How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface (2016, methodology): “Piech and colleagues [19] were able to improve accuracy by 30% using algorithms that adjust grades for grader bias and reliability.”
Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence (2015, background): “(The number ﬁve is inspired by practical systems (Wang et al., 2011; Piech et al., 2013).)”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “ML models can help in educational settings in a variety of ways: e.g., assisting in grading (Piech et al., 2013; Shermis, 2014; Kulkarni et al., 2014; Taghipour and Ng, 2016), estimating student knowledge and ability (Desmarais and Baker, 2012; Wu et al., 2020), identifying students who need…”