Toxicity Detection can be Sensitive to the Conversational Context
By Alexandros Xenos, John Pavlopoulos, I. Androutsopoulos, Lucas Dixon, Jeffrey S. Sorensen, Leo Laugier. 2021
10 citations recorded by the source corpus.
User posts whose perceived toxicity depends on the conversational context are rare in current toxicity detection datasets. Hence, toxicity detectors trained on existing datasets will also tend to disregard context, making the detection of context-sensitive toxicity harder when it does occur. We construct and publicly release a dataset of 10,000 posts with two kinds of toxicity labels: (i) annotators considered each post with the previous one as context; and (ii) annotators had no additional context. Based on this, we introduce a new task, context sensitivity estimation, which aims to identify posts whose perceived toxicity changes if the context (previous post) is also considered. We then evaluate machine learning systems on this task, showing that classifiers of practical quality can be developed, and we show that data augmentation with knowledge distillation can improve the performance further. Such systems could be used to enhance toxicity detection datasets with more context-dependent posts, or to suggest when moderators should consider the parent posts, which often may be unnecessary and may otherwise introduce significant additional cost.
Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yulia Tsvetkov
- Vidhisha Balachandran
- Sachin Kumar
- Antonios Anastasopoulos
- Lucille Njoo
- Mohit Bansal
- Anjalie Field
- Hinrich Schütze
In Information Systems
In Natural Language Processing
- Yulia Tsvetkov
- Sachin Kumar
- Vidhisha Balachandran
- Antonios Anastasopoulos
- Lucille Njoo
- Anjalie Field
- Hinrich Schütze
- Mohit Bansal
In Safety, Robustness & Interpretability
Citations in the corpus
- Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “For toxicity detection, factors like region, political views, and social and cultural background of the users affect whether they will perceive some text as toxic (Xenos et al., 2021).”