Ex Machina: Personal Attacks Seen at Scale

By Ellery Wulczyn, Nithum Thain, Lucas Dixon. 2016

846 citations recorded by the source corpus.

The damage personal attacks cause to online discourse motivates many platforms to try to curb the phenomenon. However, understanding the prevalence and impact of personal attacks in online platforms at scale remains surprisingly difficult. The contribution of this paper is to develop and illustrate a method that combines crowdsourcing and machine learning to analyze personal attacks at scale. We show an evaluation method for a classifier in terms of the aggregated number of crowd-workers it can approximate. We apply our methodology to English Wikipedia, generating a corpus of over 100k high quality human-labeled comments and 63M machine-labeled ones from a classifier that is as good as the aggregate of 3 crowd-workers, as measured by the area under the ROC curve and Spearman correlation. Using this corpus of machine-labeled scores, our methodology allows us to explore some of the open questions about the nature of online personal attacks. This reveals that the majority of personal attacks on Wikipedia are not the result of a few malicious users, nor primarily the consequence of allowing anonymous contributions from unregistered users.

Subfields: Artificial Intelligence, Communication, Information Systems, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Communication

In Information Systems

In Safety, Robustness & Interpretability

Citations in the corpus

Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, background): “, 2019), wiki toxicity (Wulczyn et al., 2017) and IMDb reviews (Maas et al.”
Challenges in Detoxifying Language Models (2021, methodology): “The score indicates fair overall agreement, and is comparable to the level of agreement reported in prior work (Ross et al., 2016; Wulczyn et al., 2017).”
The Risk of Racial Bias in Hate Speech Detection (2019, background): “Many datasets have been created, but most are either small-scale pilots ( ∼ 100 instances; Kwok and Wang, 2013; Burnap and Williams, 2015; Zhang et al., 2018), or focus on other domains (e.g., Wikipedia edits; Wulczyn et al., 2017).”
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “While most research has concentrated on single-sentence toxic content, including explicit insults (Wulczyn et al., 2017; David-son et al., 2017; Zampieri et al., 2019; Rosen-thal et al., 2021) or more covert offenses (Wang and Potts, 2019; Breitfeller et al., 2019; Han and Tsvetkov, 2020; Price et…”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “For offensive language detection, inter-annotator agreement (IAA) on human labeling tasks is typically low (Fortuna, 2017; Wulczyn et al., 2017).”
Recipes for Safety in Open-domain Chatbots (2020, methodology): “Classiﬁers trained with different truncated dialogue lengths perform almost equally on WTC, S and BBF and BAD.”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “The datasets Wikipedia Toxic Comments dataset (WTC) (Wul- czyn et al., 2017), Build-It Break-It Fix-It (BBF) (Dinan et al., 2019a) and Bot Adversarial Dialogue dataset (BAD) (Xu et al., 2020) are used to train a binary classifier (safe or not safe) given the dialogue context as input.”
Bot-Adversarial Dialogue for Safe Conversational Agents (2021, methodology): “These tasks include various combinations of the Wikipedia Toxic Comments dataset (WTC) (Wulczyn et al., 2017), Standard (S) and adversarial Build-it, Break-it, Fix-it (BBF) data from Dinan et al. (2019a), as well as semi-supervised data created from labeling the pushshift.io Reddit (Baumgartner et a”
Using In-Context Learning to Improve Dialogue Safety (2023, methodology): “Concretely, we use Wikipedia Toxic Comments (Wulczyn et al., 2017) and the safety data from Dinan et al.”
Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection (2021, background): “Models can also misclassify interpersonal abuse and incivil language as hateful (Wulczyn et al., 2017a; Zampieri et al., 2019; Palmer et al., 2020).”
Social Bias Frames: Reasoning about Social and Power Implications of Language (2019, methodology): “Most dataset creation work has cast this detection problem as binary classification (Waseem and Hovy, 2016; Wulczyn et al., 2017; Davidson et al., 2017; Founta et al., 2018), Recently, Zampieri et al.”
Reducing Gender Bias in Abusive Language Detection (2018, background): “So far, many efforts were put into deﬁning and constructing abusive language datasets from different sources and labeling them through crowd-sourcing or user moderation (Waseem and Hovy, 2016; Waseem, 2016; Founta et al., 2018; Wulczyn et al., 2017).”
Challenges and frontiers in abusive content detection (2019, background): “Similarly, in Wulczyn et al.’s dataset of over 100,000 Wikipedia comments, ‘toxicity’ is defined in relation to how likely it is to make individuals leave a discussion (Wulczyn et al., 2017).”
Challenges for Toxic Comment Classification: An In-Depth Error Analysis (2018, background): “Besides looking at toxicity of online comments (Wulczyn et al., 2017; Georgakopoulos et al., 2018), related research includes the investigation of hate speech (Badjatiya et al.”
Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification (2019, methodology): “ingful results. The full list of identities labeled can be found in Figure 4. This data was also labeled for toxicity using the same crowd rating guidelines as published by the Perspective API ([18], [19]). This labeling asks raters to rate the toxicity of a comment, selecting from “Very Toxic”, “To”
Understanding Abuse: A Typology of Abusive Language Detection Subtasks (2017, background): “Wulczyn et al. (2017) annotates for personal attacks, which likely encompasses identifying cyberbullying, hate speech, and offensive language.”
Deceiving Google's Perspective API Built for Detecting Toxic Comments (2017, background): “As a result, there has been many calls for researchers to develop methods to automatically detect the abusive or toxic context in the real time [6].”