But Who Protects the Moderators? The Case of Crowdsourced Image Moderation

By Brandon Dang, Martin J. Riedl, Matthew Lease. 2018

30 citations recorded by the source corpus.

Though detection systems have been developed to identify obscene content such as pornography and violence, artificial intelligence is simply not good enough to fully automate this task yet. Due to the need for manual verification, social media companies may hire internal reviewers, contract specialized workers from third parties, or outsource to online labor markets for the purpose of commercial content moderation. These content moderators are often fully exposed to extreme content and may suffer lasting psychological and emotional damage. In this work, we aim to alleviate this problem by investigating the following question: How can we reveal the minimum amount of information to a human reviewer such that an objectionable image can still be correctly identified? We design and conduct experiments in which blurred graphic and non-graphic images are filtered by human moderators on Amazon Mechanical Turk (AMT). We observe how obfuscation affects the moderation experience with respect to image classification accuracy, interface usability, and worker emotional well-being.

Subfields: Artificial Intelligence, Law, Sociology and Political Science, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Law

In Sociology and Political Science

In AI Ethics, Fairness & Society

Citations in the corpus

Challenges in Detoxifying Language Models (2021, background): “An important consideration for human evaluation is that the annotation process itself can impose emotional burden on annotators exposed to toxic content (Dang et al., 2018; Steiger et al., 2021).”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, methodology): “The production of harmful content is currently mitigated using combinations of four methods: data sourcing [30]; human moderation of content included in training data [65]; automated moderation of content included in training data [101]; and keyword deny-lists [149].”
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, methodology): “These include versions of the Positive and Negative Affect Schedule (PANAS) [56] used in [15, 16, 31] and the Scale of Positive and Negative Experience (SPANE) [17] used in [15, 16].”
Improving alignment of dialogue agents via targeted human judgements (2022, background): “Well-being surveys Exposing annotators to harmful content could negatively impact their well-being (Dang et al., 2018; Steiger et al., 2021).”
On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research (2023, unknown): “Human annotation is not only increasingly expensive but also poses a serious mental health risk to evaluators exposed to highly toxic content, leaving them vulnerable to lasting psychological harm (Dang et al., 2018; Steiger et al., 2021).”
Quantifying the Invisible Labor in Crowd Work (2021, background): “For instance, crowd workers might label content so that Facebook’s News Feed algorithm will not recommend posts that are filled with hate speech or pedophilia [17, 46].”
Testing Stylistic Interventions to Reduce Emotional Impact of Content Moderation Workers (2019, background): “Dang, Riedl, and Lease 2018, have experimented with blurred images with varying degrees of blur in a simulated review environment.”
Fast, Accurate, and Healthier: Interactive Blurring Helps Moderators Reduce Exposure to Harmful Content (2020, methodology): “Our earlier work (Dang, Riedl, and Lease 2018) is the ﬁrst we are aware of to propose blurring imagery to reduce moderator exposure to disturbing content.”
The Psychological Well-Being of Content Moderators: The Emotional Labor of Commercial Moderation and Avenues for Improving Support (2021, background): “[36]’s study by evaluating their interactive blurring tool.”