Conversations Gone Awry: Detecting Early Signs of Conversational Failure

By Justine Zhang, Cristian Danescu-Niculescu-Mizil, Cristian Danescu-Niculescu-Mizil, Lucas Dixon, Yiqing Hua, Nithum Thain, Dario Taraborelli. 2018

223 citations recorded by the source corpus.

One of the main challenges online social systems face is the prevalence of antisocial behavior, such as harassment and personal attacks. In this work, we introduce the task of predicting from the very start of a conversation whether it will get out of hand. As opposed to detecting undesirable behavior after the fact, this task aims to enable early, actionable prediction at a time when the conversation might still be salvaged. To this end, we develop a framework for capturing pragmatic devices—such as politeness strategies and rhetorical prompts—used to start a conversation, and analyze their relation to its future trajectory. Applying this framework in a controlled setting, we demonstrate the feasibility of detecting early warning signs of antisocial behavior in online discussions.

Subfields: Artificial Intelligence, General Health Professions, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Health Professions

In Natural Language Processing

Citations in the corpus

RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, background): “The creation of REALTOXICITYPROMPTS was partly inspired by work in detecting conversational patterns that can cause derailment into antisocial behavior in online conversations (Zhang et al., 2018; Stoop et al., 2019; Karan and Šnajder, 2019).”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, background): “[170] addressed the problem of predicting from the very beginning of a conversation whether it will get out of hand.”
Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts (2021, background): “To further strengthen dialogue safety, future research on detection of offensive context (Dinan et al., 2019a; Zhang et al., 2018a) and subsequent generation of nonprovocative counter-speech (Chung et al., 2019) is crucial.”
Context Sensitivity Estimation in Toxicity Detection (2021, background): “What most of these systems have in common, besides aiming to promote healthy discussions online (Zhang et al., 2018), is that they disregard the conversational context (e.g., the parent post in the discussion), making the detection of context-sensitive toxicity a lot harder.”
Time Waits for No One! Analysis and Challenges of Temporal Misalignment (2021, background): “Though not studied here, the same issues may arise in annotation efforts; consider, for example, recent work on controversy (Zhang et al., 2018) and social norms (Xu et al., 2021; Zhou et al., 2021) likely hinges on constructs that may be time sensitive.”
Crowdsourcing Subjective Tasks: The Case Study of Understanding Toxicity in Online Discussions (2019, background): “As [47] points out, much of this work builds on existing machine learning approaches in fields like sentiment analysis [5, 35, 50, 52] and spam detection [37, 39].”
Multi-Modal Open-Domain Dialogue (2020, methodology): “…a well-studied, but still unsolved, research area (Dinan et al., 2019b; Liu et al., 2019; Dinan et al., 2019a; Blod-gett et al., 2020; Khatri et al., 2018; Sch¨afer and Burtenshaw, 2019; Zhang et al., 2018a), yet we note that safety in the context of image-dialogue is relatively less explored.”
Civil Rephrases Of Toxic Texts With Self-Supervised Transformers (2021, background): “Then, a decoder, aware of the destination attribute, generated a transferred sentence (Shen et al., 2017; Hu et al., 2017; Fu et al., 2018; Zhang et al., 2018c; Xu et al., 2018; John et al., 2019).”
Toxicity Detection can be Sensitive to the Conversational Context (2021, background): “What most of these systems have in common, besides aiming to promote healthy discussions online (Zhang et al., 2018), is that they disregard much of ∗Corresponding author. the conversational context, making the detection of context-sensitive toxicity a lot harder.”
Something’s Brewing! Early Prediction of Controversy-causing Posts from Discussion Features (2019, background): “…comment volume/popularity/structure (Szabo and Huberman, 2010; Kim et al., 2011; Tatar et al., 2011; Backstrom et al., 2013; He et al., 2014; Zhang et al., 2018b) , or eventual post article score (Rangwala and Jamali, 2010; Szabo and Huberman, 2010), ; but all where, like us , the paradigm…”
Can Large Language Models Transform Computational Social Science? (2023, methodology): “Using the Conversations Gone Awry corpus (Zhang et al., 2018), we investigate whether LLMs can predict future toxicity from early cues.”
Robust Conversational Agents against Imperceptible Toxicity Triggers (2022, background): “In addition, there is a body of work in detecting toxic behavior in conversational agents (Zhang et al., 2018; Almerekhi et al., 2019; Baheti et al., 2021) that can be utilized to design ethically aligned systems.”
Preemptive Toxic Language Detection in Wikipedia Comments Using Thread-Level Context (2019, methodology): “At ﬁrst we considered using the dataset from (Zhang et al., 2018a), but found it rather small ( ∼ 1200 examples) for deep learning approaches.”
Empirical Analysis of Multi-Task Learning for Reducing Model Bias in Toxic Comment Detection (2019, methodology): “Current research involves investigating common challenges in toxic comment classiﬁcation (van Aken et al. 2018), identifying subtle forms of toxicity (Noever 2018), detecting early signs of toxicity (Zhang et al. 2018), and analysing sarcasm within conversations (Ghosh, Fabbri, and Muresan 2018).”
Detecting harassment in real-time as conversations develop (2019, background): “Zhang et al. (2018b) conduct an extensive evaluation of approaches for detecting hate speech so far and propose a combination of CNNs and RNNs to outperform them.”
A Just and Comprehensive Strategy for Using NLP to Address Online Abuse (2019, background): “Recent work has shown that it is possible to predict whether a conversation will become toxic on Wikipedia (Zhang et al., 2018) and whether bullying will occur on Instagram (Liu”