Challenges and frontiers in abusive content detection

By Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott A. Hale, Helen Z. Margetts. 2019

249 citations recorded by the source corpus.

Online abusive content detection is an inherently difficult task. It has received considerable attention from academia, particularly within the computational linguistics community, and performance appears to have improved as the field has matured. However, considerable challenges and unaddressed frontiers remain, spanning technical, social and ethical dimensions. These issues constrain the performance, efficiency and generalizability of abusive content detection systems. In this article we delineate and clarify the main challenges and frontiers in the field, critically evaluate their implications and discuss potential solutions. We also highlight ways in which social scientific insights can advance research. We discuss the lack of support given to researchers working with abusive content and provide guidelines for ethical research.

Subfields: Artificial Intelligence, Information Systems, Sociology and Political Science, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Sociology and Political Science

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

HateCheck: Functional Tests for Hate Speech Detection Models (2020, methodology): “This is motivated by previous work, which highlights syntactic relationships and long-range dependencies as model weak points (Burnap and Williams, 2015; Vidgen et al., 2019).”
Social Biases in NLP Models as Barriers for Persons with Disabilities (2020, background): “It is important to recognize that social norms around language are contextual and differ across groups (Castelle, 2018; Davidson et al., 2019; Vidgen et al., 2019).”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “Several surveys cover approaches to this problem (Schmidt & Wiegand, 2017; Fortuna & Nunes, 2018; Vidgen et al., 2019), and there exist reviews of offensive language datasets (Fortuna et al., 2020; Vidgen & Derczynski, 2020).”
Recipes for Safety in Open-domain Chatbots (2020, background): “Vidgen et al. (2019) recently surveyed work in online abusive content detection.”
Mitigating harm in language models with conditional-likelihood filtration (2021, methodology): “Deﬁning undesirable text is difﬁcult and context-dependent; using automated methods to ﬂag it presents additional challenges [Vidgen et al., 2019].”
Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets (2021, background): “[37] describe how computational methods to robustly detect and measure abusive, harmful content are unsolved research and community challenges.”
Improving alignment of dialogue agents via targeted human judgements (2022, background): “Prior work has argued that broad umbrella terms like “toxicity” can be ambiguous (Banko et al., 2020; Vidgen et al., 2019; Welbl et al., 2021) so we base our rules on more fine-grained definitions including the online harmful language taxonomy from Banko et al. (2020), and definitions of…”
Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection (2021, methodology): “In addition, many datasets are formed with bootstrapped sampling, such as keyword searches, due to the low prevalence of hate speech ‘in the wild’ (Vidgen et al., 2019b).”
Dynabench: Rethinking Benchmarking in NLP (2021, background): “Few high-quality, varied and large training datasets are available for training hate detection systems (Vidgen and Derczynski, 2020; Poletto et al., 2020; Vidgen et al., 2019).”
Directions in abusive language training data, a systematic review: Garbage in, garbage out (2020, background): “Other authors have sought to deductively theorise additional categories, such as ‘concept-directed’ abuse, although these have not been widely adopted [11].”
Towards Ethics by Design in Online Abusive Content Detection (2020, background): “Viewing and annotating abusive content for prolonged periods of time can cause signiﬁcant distress to human annotators (Roberts, 2016; Vidgen et al., 2019).”
Detecting East Asian Prejudice on Social Media (2020, methodology): “The field of hate speech detection sits at the intersection of social science and computer science, and is fraught with not only technical challenges but also deep-routed ethical and theoretical considerations [30].”
Recalibrating classifiers for interpretable abusive content detection (2020, result): “These low to moderate levels of agreement are in line with other annotation studies of abusive content, reﬂecting the difﬁculty of such tasks (Vidgen et al., 2019).”
Towards multidomain and multilingual abusive language detection: a survey (2021, background): “It has been stated that ensuring that a model can detect abusive language across different domains is one of the main challenges and an important frontier [137].”
Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective (2020, background): “Vidgen et al. (2019) extended this topology to three dimensions: (1) whether the abuse is directed towards an individual, an identity (based on belonging to a demographic category, social group, or organization), or concept (such as a belief system, country or ideology), (2) who receives the abuse…”