Context Sensitivity Estimation in Toxicity Detection

By Alexandros Xenos, John Pavlopoulos, Ion Androutsopoulos. 2021

User posts whose perceived toxicity depends on the conversational context are rare in current toxicity detection datasets. Hence, toxicity detectors trained on current datasets will also disregard context, making the detection of context-sensitive toxicity a lot harder when it occurs. We constructed and publicly release a dataset of 10k posts with two kinds of toxicity labels per post, obtained from annotators who considered (i) both the current post and the previous one as context, or (ii) only the current post. We introduce a new task, context-sensitivity estimation, which aims to identify posts whose perceived toxicity changes if the context (previous post) is also considered. Using the new dataset, we show that systems can be developed for this task. Such systems could be used to enhance toxicity detection datasets with more context-dependent posts or to suggest when moderators should consider the parent posts, which may not always be necessary and may introduce additional costs.

Subfields: Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “For toxicity detection, factors like region, political views, and social and cultural background of the users affect whether they will perceive some text as toxic (Xenos et al., 2021).”
Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts (2021, background): “On the other hand, Pavlopoulos et al. (2020); Xenos et al. (2021) created labeled datasets for toxicity detection in single turn conversations and studied context-sensitivity in detection models.”
On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark (2021, background): “To the best of our knowledge, context-sensitive unsafety is only brieﬂy discussed in a few past works (Xenos et al., 2021; Baheti et al., 2021), as summarized in Table 1.”