Detecting Offensive Content in Open-domain Conversations using Two Stage Semi-supervision

By Chandra Khatri, Behnam Hedayatnia, Rahul Goel, Anu Venkatesh, Raefer Gabriel, Arindam Mandal. 2018

33 citations recorded by the source corpus.

As open-ended human-chatbot interaction becomes commonplace, sensitive content detection gains importance. In this work, we propose a two stage semi-supervised approach to bootstrap large-scale data for automatic sensitive language detection from publicly available web resources. We explore various data selection methods including 1) using a blacklist to rank online discussion forums by the level of their sensitiveness followed by randomly sampling utterances and 2) training a weakly supervised model in conjunction with the blacklist for scoring sentences from online discussion forums to curate a dataset. Our data collection strategy is flexible and allows the models to detect implicit sensitive content for which manual annotations may be difficult. We train models using publicly available annotated datasets as well as using the proposed large-scale semi-supervised datasets. We evaluate the performance of all the models on Twitter and Toxic Wikipedia comments testsets as well as on a manually annotated spoken language dataset collected during a large scale chatbot competition. Results show that a model trained on this collected data outperforms the baseline models by a large margin on both in-domain and out-of-domain testsets, achieving an F1 score of 95.5% on an out-of-domain testset compared to a score of 75% for models trained on public datasets. We also showcase that large scale two stage semi-supervision generalizes well across multiple classes of sensitivities such as hate speech, racism, sexual and pornographic content, etc. without even providing explicit labels for these classes, leading to an average recall of 95.5% versus the models trained using annotated public datasets which achieve an average recall of 73.2% across seven sensitive classes on out-of-domain testsets.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

Recipes for Safety in Open-domain Chatbots (2020, methodology): “We consider the Wikipedia Toxic Comments dataset (WTC) (Wulczyn et al., 2017) designed to identify personal attacks online, consisting of ∼ 150k examples; we use the version that treats the data as a two-class problem (Khatri et al., 2018a; Dinan et al., 2019c).”
Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack (2019, methodology): “In the same way as in (Khatri et al., 2018), every label except non-toxic is grouped into a class OFFENSIVE while the non-toxic class is kept as the SAFE class.”
Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prize (2018, methodology): “More information about the model can be obtained at (Khatri et al., 2018b).”
ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI (2021, methodology): “The majority of currently deployed systems use simple keyword spotting techniques, (e.g. Ram et al., 2018; Khatri et al., 2018), which tend to produce a high number of false positives, such as cases in which the user expresses frustration, or use of profanities for emphasis, as well as false…”
Further Advances in Open Domain Dialog Systems in the Third Alexa Prize Socialbot Grand Challenge (2020, methodology): “…scientific advances and reduce time invested into infrastructure, hosting and scaling complex model-driven bots, we created CoBot, a conversational bot toolkit in Python for natural language understanding and dialog management, as described in detail in Khatri et al. (2018) and shown in Figure 1.”
“Nice Try, Kiddo”: Investigating Ad Hominems in Dialogue Responses (2021, background): “For analysis, Sheng et al. (2019) evaluate societal biases in language generation, Curry and Rieser (2018) study how conversational systems respond to sexual harassment, and Khatri et al. (2018) detect offensive content with a semi-supervised approach.”
Directions in Abusive Language Training Data: Garbage In, Garbage Out (2020, background): “The challenges posed by small datasets could potentially be overcome through machine learning techniques such as ‘semi-supervised’ and ‘active’ learning [66], although these have only been limitedly applied to abusive content detection so far [65].”
Directions in abusive language training data, a systematic review: Garbage in, garbage out (2020, background): “The challenges posed by small datasets could potentially be overcome through machine learning techniques such as ‘semi-supervised’ and ‘active’ learning [93], although these have only been limitedly applied to abusive content detection so far [94].”
Multi-Modal Open-Domain Dialogue (2020, background): “…is indeed a well-studied, but still unsolved, research area (Dinan et al., 2019b; Liu et al., 2019; Dinan et al., 2019a; Blod-gett et al., 2020; Khatri et al., 2018; Sch¨afer and Burtenshaw, 2019; Zhang et al., 2018a), yet we note that safety in the context of image-dialogue is relatively less…”