Dialogue Natural Language Inference

By Sean Welleck, Jason Aaron Edward Weston, Arthur Szlam, Kyunghyun Cho. 2018

270 citations recorded by the source corpus.

Consistency is a long standing issue faced by dialogue models. In this paper, we frame the consistency of dialogue agents as natural language inference (NLI) and create a new natural language inference dataset called Dialogue NLI. We propose a method which demonstrates that a model trained on Dialogue NLI can be used to improve the consistency of a dialogue model, and evaluate the method with human evaluation and with automatic metrics on a suite of evaluation sets designed to measure a dialogue model’s consistency.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, background): “Since penalizing only the upcoming token limits prefix strings to only those found in the training distribution, it can cause a distribution mismatch between training sequences and generated sequences (Welleck et al., 2019a).”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “Altering the behavior of the generation method can dramatically influence human evaluations, while maintaining identical or near-identical perplexity (See et al., 2019; Welleck et al., 2020, 2019; Adiwardana et al., 2020; Roller et al., 2020).”
Measuring Attribution in Natural Language Generation Models (2021, methodology): “These have been measured via dialogue-specific natural language inference methods, often via human studies and data creation (Welleck et al. 2019; Mehri and Eskenazi 2020; Gupta et al. 2021; Honovich et al. 2021; Dziri et al. 2021; Santhanam et al. 2021).”
On Faithfulness and Factuality in Abstractive Summarization (2020, background): “…also report on ROUGE (Lin and Hovy, 2003) scores, BERTScore (Zhang et al., 2020) and semantic inference metric such as textual entailment (Pasunuru and Bansal, 2018; Welleck et al., 2019; Falke et al., 2019; Kryscinski et al., 2019b) and question answering (Arumae and Liu, 2019; Wang et al., 2020).”
Survey of Hallucination in Natural Language Generation (2022, background): “Two possible types of inconsistency occur in open-domain dialogue generation: (1) self-inconsistency [147, 159] or incoherence [8, 29] among the system utterances, such as when the system contradicts its previous utterance; (2) external inconsistency with an external source, such as factually incorr”
The Second Conversational Intelligence Challenge (ConvAI2) (2019, background): “One possible solution to these problems is the use of dialogue natural language inference (NLI) [12], a new task that has been proposed that evaluates exactly these problems.”
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries (2020, background): “6 Re-ranking with QAGS Several works explore the use of natural language inference (NLI) models to detect factual consistency in generated text (Welleck et al., 2019; Falke et al., 2019).”
Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training (2019, background): “That is, they can produce logically or factually inaccurate, or contradicting statements (Welleck et al., 2018; Zhang et al., 2018; Hayashi et al., 2019; Petroni et al., 2019).”
Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameters (2021, methodology): “It is also noteworthy that on the SNLI, Dialogue NLI, and SciTail datasets, all the PHM-LSTM variants outperform the standard LSTM model.”
Personalizing Dialogue Agents via Meta-Learning (2019, methodology): “Aside of standards evaluation metrics, we also train a Natural Language Inference (NLI) model using Dialog NLI (Sean et al., 2018) dataset, a recently proposed corpus based on Persona dataset, with NLI annotation between persona description sentences and dialogues utterance.”
DynaEval: Unifying Turn and Dialogue Level Evaluation (2021, background): “…human conversations in the sense that they lack certain conversation-level attributes, such as coherence (Cervone et al., 2018), consistency (Welleck et al., 2019; Nie et al., 2020), diversity (Li et al., 2016; Wu et al., 2020) and engagement (Ghande-harioun et al., 2019; Ghazarian et al.,…”
Natural Language Inference in Context - Investigating Contextual Reasoning over Long Texts (2020, methodology): “Following the NLI benchmark setting (Bowman et al. 2015; Williams, Nangia, and Bowman 2018; Welleck et al. 2019), we employ the overall accuracies as the main evaluation method.”
Evaluating Coherence in Dialogue Systems using Entailment (2019, methodology): “Recently, Welleck et al. (Welleck et al., 2018) investigated the use of NLI models (e.g., ESIM (Chen et al., 2016) and In-ferSent (Conneau et al., 2017)) to measure consistency in dialogue systems.”
Improving Factual Consistency Between a Response and Persona Facts (2021, methodology): “Besides the standard test set, which was annotated by one crowd-worker , there is Test Gold containing 12 , 376 of test pairs, which were annotated by three crowd-workers (Welleck et al., 2019).”
Automatically Exposing Problems with Neural Dialog Models (2021, background): “In terms of consistency detection, Dziri et al. (2019); Welleck et al. (2019); Li et al. (2020) introduce and suggest using natural language inference to model conversation coherence.”
Entity-level Factual Consistency of Abstractive Text Summarization (2021, background): “Several authors have pointed out the problem of factual inconsistency in abstractive summarization models (Kryscinski et al., 2019; Kry´sci´nski et al., 2019; Cao et al., 2018; Welleck et al., 2019).”
PERSONACHATGEN: Generating Personalized Dialogues using GPT-3 (2022, methodology): “We used a BART-large (Lewis et al., 2019) model trained on the MNLI dataset (Williams et al., 2017).”