Detecting ‘Dirt’ and ‘Toxicity’: Rethinking Content Moderation as Pollution Behaviour

By Nanna Bonde Thylstrup, Zeerak Talat. 2020

"Dirt" and "toxicity" have become established ways of understanding and describing harmful content online. While the concepts have becoming naturalised as a way of expressing harm, its uses are also mobilised to very different ends. We wish to use this article to explore the role the notions of "toxic" and "dirt" have come to play in online content moderation, and what the politics of this discourse implies. Our overarching aim is to ask how content moderation infrastructures define toxic and dirty content and what the politics of these definitions are. We argue that content moderation’s historical reliance on binary categories – and those categories’ ongoing entanglements with social systems of racism and patriarchy – embeds the infrastructures in structures that risk reproducing inequalities.

Subfields: Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Sensitive topics also vary by culture and can include hate speech, which itself is contingent on cultural norms of acceptability [242].”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “Thylstrup & Waseem (2020) caution that resorting to binary labels in itself incurs its own risk of reproducing inequalities.”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “…situations, by directly generating toxic or otherwise harmful content, by agreeing with offensive statements uttered by the conversation partner (Dinan et al., 2022), or by responding defensively or dismissively when provided with corrective feedback by the conversation partner (Ung et al., 2021).”
Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022, background): “To this effect, we should be cognizant of the existing linguistic discrimination present in our societies [118] and be careful not to inadvertently replicate them [132, 133].”
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings (2022, background): “monolingual and encode white respectability politics (Thylstrup and Talat, 2020; Kerrison et al., 2018) onto minoritized variants of English (Gehman et al.”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, background): “Thylstrup and Waseem (2020) caution that using binary labels in itself incurs the risk of reproducing inequalities.”
Context Sensitivity Estimation in Toxicity Detection (2021, background): “These posts are called toxic (Borkan et al., 2019) or abusive (Thylstrup and Waseem, 2020), and systems detecting them (Waseem and Hovy, 2016; Pavlopoulos et al., 2017b; Badjatiya et al., 2017) are called toxicity (or abusive language) detection systems.”
Toxicity Detection can be Sensitive to the Conversational Context (2021, background): “, 2020) or abusive (Thylstrup and Waseem, 2020), and systems detecting them (Waseem and Hovy, 2016; Pavlopoulos et al.”