I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling

By Yixin Nie, Mary Williamson, Mohit Bansal, Douwe Kiela, Jason Aaron Edward Weston. 2020

94 citations recorded by the source corpus.

To quantify how well natural language understanding models can capture consistency in a general conversation, we introduce the DialoguE COntradiction DEtection task (DECODE) and a new conversational dataset containing both human-human and human-bot contradictory dialogues. We show that: (i) our newly collected dataset is notably more effective at providing supervision for the dialogue contradiction detection task than existing NLI data including those aimed to cover the dialogue domain; (ii) Transformer models that explicitly hinge on utterance structures for dialogue contradiction detection are more robust and generalize well on both analysis and out-of-distribution dialogues than standard (unstructured) Transformers. We also show that our best contradiction detection model correlates well with human judgments and further provide evidence for its usage in both automatically evaluating and improving the consistency of state-of-the-art generative chatbots.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (2022, methodology): “Predicting reply corrections To obtain the corrections, we adopt a reranking-based learning method widely used in many previous studies (Nie et al., 2020; Nakano et al., 2021; Askell et al., 2021) to score and rank the generations.”
Director: Generator-Classifiers For Supervised Language Modeling (2022, positive): “The LM head of the D IRECTOR model is further ﬁne-tuned using the Blended Skill Talk (BST) tasks (Smith et al., 2020b) and the classiﬁer head is trained using the LTR version of the DECODE (Nie et al., 2020) dataset.”
Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning (2021, background): “, 2020), dialogue systems (for control, coherence and safety (Welleck et al., 2018; Smith et al., 2020; Nie et al., 2020; Xu et al., 2020a)), and instruction following (Kurita & Cho, 2020).”
DialGuide: Aligning Dialogue Model Behavior with Developer Guidelines (2022, background): “Response entailment (Welleck et al., 2019; Nie et al., 2021; Gupta et al., 2022c) predicts whether a response entails a given premise.”
Automatically Exposing Problems with Neural Dialog Models (2021, methodology): “Similarly, Nie et al. (2021) asks Mechanical Turkers to manually write contradicting dialogs for both humans and bots, or to interact with chatbots, where a frequent strategy is to ask factual questions intentionally leading to contradiction (e.g. ask “do you speak Spanish” after the bot says “I am…”
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022, background): “Though other dimensions, such as consistency (Nie et al., 2021), inquisitiveness (See et al.”
DynaEval: Unifying Turn and Dialogue Level Evaluation (2021, background): “…human conversations in the sense that they lack certain conversation-level attributes, such as coherence (Cervone et al., 2018), consistency (Welleck et al., 2019; Nie et al., 2020), diversity (Li et al., 2016; Wu et al., 2020) and engagement (Ghande-harioun et al., 2019; Ghazarian et al., 2020).”
What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation (2022, background): “∗Work done while Sarik Ghazarian was an intern at Amazon Alexa AI contradiction (Pang et al., 2020; Nie et al., 2021) amongst others.”
PERSONACHATGEN: Generating Personalized Dialogues using GPT-3 (2022, methodology): “To calculate the contra-diction score between all pairs { ( p i , p j ) } i =50 ,j =51 i =1 ,j =2 , we leveraged the dialogue contradiction detection (D ECODE ) task (Nie et al., 2020), which determines whether the previous utterance is inconsistent with any previous utterances.”
Evaluating Very Long-Term Conversational Memory of LLM Agents (2024, background): “…employs lexical overlap (Papineni et al., 2002) and semantic overlap (Zhang et al., 2019) between ground truth and the agent response, or consistency (Ghazarian et al., 2022), contradiction (Nie et al., 2021; Welleck et al., 2019), and empathy (Zhang et al., 2021a, 2022) of the agent response.”
Commonsense-Focused Dialogues for Response Generation: An Empirical Study (2021, background): “, 2018), consistency (Nie et al., 2020), personality (Zhang et al.”
Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation (2021, background): “A growing number of recent works have proposed aspect-based metrics for popular tasks such as summarization (Kry´sci´nski et al., 2019; Wang et al., 2020) and dialog (Mehri and Eskenazi, 2020b; Nie et al., 2020).”
DialFact: A Benchmark for Fact-Checking in Dialogue (2021, methodology): “…claims using existing state-of-298 the-art models related to our task: namely Dialogue299 NLI (Welleck et al., 2019), Dialogue contradiction300 detection (Nie et al., 2021), FEVER based fact ver-301 ification (Schuster et al., 2021) and fact-checking302 on colloquial claims (Kim et al., 2021).”
TRUE: Re-evaluating Factual Consistency Evaluation (2022, methodology): “…and methods that target a speciﬁc task or domain, such as summarization (Falke et al., 2019; Kryscinski et al., 2020; Wang et al., 2020; Scialom et al., 2021; Deutsch et al., 2021; Xie et al., 2021) or dialogue (Dziri et al., 2021; Honovich et al., 2021; Nie et al., 2021; Qin et al., 2021).”