Like Trainer, Like Bot? Inheritance of Bias in Algorithmic Content Moderation

By Reuben Binns, Michael Veale, M. V. Kleek, Nigel R. Shadbolt. 2017

180 citations recorded by the source corpus.

The internet has become a central medium through which `networked publics' express their opinions and engage in debate. Offensive comments and personal attacks can inhibit participation in these spaces. Automated content moderation aims to overcome this problem using machine learning classifiers trained on large corpora of texts manually annotated for offence. While such systems could help encourage more civil debate, they must navigate inherently normatively contestable boundaries, and are subject to the idiosyncratic norms of the human raters who provide the training data. An important objective for platforms implementing such measures might be to ensure that they are not unduly biased towards or against particular norms of offence. This paper provides some exploratory methods by which the normative biases of algorithmic content moderation systems can be measured, by way of a case study using an existing dataset of comments labelled for offence. We train classifiers on comments labelled by different demographic subsets (men and women) to understand how differences in conceptions of offence between these groups might affect the performance of the resulting models on various test sets. We conclude by discussing some of the ethical choices facing the implementers of algorithmic moderation systems, given various desired levels of diversity of viewpoints amongst discussion participants.

Subfields: Artificial Intelligence, General Social Sciences

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

Citations in the corpus

Fairness in Machine Learning: Lessons from Political Philosophy (2017, background): “(2016); Caliskan-Islam et al. (2016). In such cases, the problem is not necessarily one of specific harms to specific members of a social group, but rather one of the way in which certain groups are represented in digital cultural artefacts, such as natural language classifiers or search engine resu”
Cyberbullying Detection With Fairness Constraints (2020, background): “BERT [28], ELMo [29], GPT-2 [30]), bias in sentence embeddings, bias in machine learning algorithms, or bias in human annotators [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42].”
Algorithmic content moderation: Technical and political challenges in the automation of platform governance (2020, background): “To the extent that automatically blocking or allowing certain types of content can be seen as distributive harms or beneﬁts conferred on protected groups, or privileging more or less ‘deserving’ individuals, they could be cast within the paradigm of algorithmic ‘fair-ness’ (Binns et al., 2017).”
Challenges and frontiers in abusive content detection (2019, background): “Binns et al. show that male and female annotators have different perceptions of what is considered toxic (Binns, Veale, Van Kleek, & Shadbolt, 2017).”
Directions in abusive language training data, a systematic review: Garbage in, garbage out (2020, background): “to investigate how the demographics of annotators impacts the annotations they make [12, 47].”
Fairer machine learning in the real world: Mitigating discrimination without collecting sensitive data (2017, background): “Only a small amount of work has been done in this space (see Veale, 2017 for one example), and we argue strongly that this should increase.”
Human-Machine Collaboration for Content Regulation: The Case of Reddit Automoderator (2019, background): “Prior research has shown that gender shapes individuals’ conceptions of offense in online posts [7].”
Content moderation, AI, and the question of scale (2020, background): “An effective tool may learn to make the same kinds of distinctions as before (Binns et al., 2017; Gehl et al., 2017).”
Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation (2022, background): “Identity effects on annotation have also been studied from the perspective of how gender can impact inter-annotator agreement, toxicity labels, and the resultant classifier performance [9].”
Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering (2022, background): “Other reviews include the analysis of ethical implications (Kiritchenko et al., 2021) and of problems such as bias replication (Binns et al., 2017; Davidson et al., 2019; Vidgen and Der-czynski, 2020; Sap et al., 2019; Tsvetkov, 2020).”
On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research (2023, unknown): “These are challenging first steps, though, because the perception of toxicity and hate-speech is known to vary among different identity groups (Goyal et al., 2022) and genders (Binns et al., 2017).”
Tackling Online Abuse: A Survey of Automated Abuse Detection Methods (2019, background): “For e.g., some recent works have investigated ways to mitigate gender bias in models (Binns et al., 2017; Park et al., 2018).”
Directions in Abusive Language Training Data: Garbage In, Garbage Out (2020, background): “For instance, Binns et al. use the detailed metadata in the datasets provided by Wulczyn et al. to investigate how the demographics of annotators impacts the annotations they make [11, 117].”