The Psychological Well-Being of Content Moderators: The Emotional Labor of Commercial Moderation and Avenues for Improving Support

By Miriah Steiger, Timir Bharucha, Sukrit Venkatagiri, Martin J. Riedl, Matthew Lease. 2021

232 citations recorded by the source corpus.

An estimated 100,000 people work today as commercial content moderators. These moderators are often exposed to disturbing content, which can lead to lasting psychological and emotional distress. This literature review investigates moderators’ psychological symptomatology, drawing on other occupations involving trauma exposure to further guide understanding of both symptoms and support mechanisms. We then introduce wellness interventions and review both programmatic and technological approaches to improving wellness. Additionally, we review methods for evaluating intervention efficacy. Finally, we recommend best practices and important directions for future research. Content Warning: we discuss the intense labor and psychological effects of CCM, including graphic descriptions of mental distress and illness.

Subfields: Artificial Intelligence, Radiological and Ultrasound Technology, Sociology and Political Science, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Safety, Robustness & Interpretability

Citations in the corpus

Challenges in Detoxifying Language Models (2021, background): “An important consideration for human evaluation is that the annotation process itself can impose emotional burden on annotators exposed to toxic content (Dang et al., 2018; Steiger et al., 2021).”
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, methodology): “We were concerned about our task causing harm to the red team, so we reviewed available literature in Trust & Safety and related ﬁelds (e.g., [26, 15, 16, 49, 5, 27, 31]), as well as conducted a series of informational interviews with industry professionals in the ﬁeld.”
Improving alignment of dialogue agents via targeted human judgements (2022, background): “Well-being surveys Exposing annotators to harmful content could negatively impact their well-being (Dang et al., 2018; Steiger et al., 2021).”
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection (2021, background): “Note that while content moderation can induce significant psyhcological harms in moderators (Roberts, 2017; Steiger et al., 2021), full automation also has significant risks.”
Decolonizing Content Moderation: Does Uniform Global Community Standard Resemble Utopian Equality or Western Power Hegemony? (2023, background): “Moreover, the platforms need to provide both mental health care (e.g., clinical care, resilience training) and technological support (e.g., tools to blur graphic content or see them in grayscale) to protect the moderators from the psychological harms of content moderation [137].”
ProsocialDialog: A Prosocial Backbone for Conversational Agents (2022, background): “For example, utterance 6 in Figure 1b is grounded in the prosocial intent to remind the inter-locutor that “You should look out for others.””
Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing (2021, background): “Although literature on the emotional toll on crowdworkers is still scant (Huws, 2015), there is growing literature on the psychological cost of work done by commercial content moderators (Steiger et al., 2021).”
Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics (2022, background): “In many situations where auditing is desirable (e.g. toxic or unsafe content screening), extensive human labour is undesirable or even unethical (Steiger et al., 2021; Shmueli et al., 2021).”