Moderating with the Mob: Evaluating the Efficacy of Real-Time Crowdsourced Fact-Checking
By William Godel, Zeve Sanderson, Kevin Aslett, Jonathan Nagler, Richard Bonneau, N. Persily, Joshua A. Tucker. 2021
Reducing the spread of false news remains a challenge for social media platforms, as the current strategy of using third-party fact- checkers lacks the capacity to address both the scale and speed of misinformation diffusion. Research on the “wisdom of the crowds” suggests one possible solution: aggregating the evaluations of ordinary users to assess the veracity of information. In this study, we investigate the effectiveness of a scalable model for real-time crowdsourced fact-checking. We select 135 popular news stories and have them evaluated by both ordinary individuals and professional fact-checkers within 72 hours of publication, producing 12,883 individual evaluations. Although we find that machine learning-based models using the crowd perform better at identifying false news than simple aggregation rules, our results suggest that neither approach is able to perform at the level of professional fact-checkers. Additionally, both methods perform best when using evaluations only from survey respondents with high political knowledge, suggesting reason for caution for crowdsourced models that rely on a representative sample of the population. Overall, our analyses reveal that while crowd-based systems provide some information on news quality, they are nonetheless limited—and have significant variation—in their ability to identify false news.
Subfields: AI Ethics, Fairness & Society
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In AI Ethics, Fairness & Society
- Alexandra Sasha Luccioni
- Irene Solaiman
- Marie-Therese Png
- Zeerak Talat
- Yacine Jernite
- Su Lin Blodgett
- Margaret Mitchell
- Sara Hooker
Citations in the corpus
- Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Mitigation and Interventions Interventions on technical systems include encouraging people to shift their attention to the accuracy of posts they might share [180], using crowd-sourced fact checking [90], and using digital forensics to detect AI-generated content [76].”