Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training
By Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, Jason Aaron Edward Weston. 2019
212 citations recorded by the source corpus.
Generative dialogue models currently suffer from a number of problems which standard maximum likelihood training does not address. They tend to produce generations that (i) rely too much on copying from the context, (ii) contain repetitions within utterances, (iii) overuse frequent words, and (iv) at a deeper level, contain logical flaws.In this work we show how all of these problems can be addressed by extending the recently introduced unlikelihood loss (Welleck et al., 2019) to these cases. We show that appropriate loss functions which regularize generated outputs to match human distributions are effective for the first three issues. For the last important general issue, we show applying unlikelihood to collected data of what a model should not do is effective for improving logical consistency, potentially paving the way to generative models with greater reasoning ability. We demonstrate the efficacy of our approach across several dialogue tasks.
Subfields: Artificial Intelligence, Natural Language Processing, Alignment & RLHF
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jason Aaron Edward Weston
- Yejin Choi
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Noah A. Smith
- Caiming Xiong
In Natural Language Processing
- Jason Aaron Edward Weston
- Mohit Bansal
- Yejin Choi
- Douwe Kiela
- Kurt Shuster
- Emily Dinan
- Y-Lan Boureau
- Eric Michael Smith
In Alignment & RLHF
- Ethan Perez
- Christopher D. Manning
- Nat McAleese
- Yejin Choi
- Douwe Kiela
- Prithviraj Ammanabrolu
- Francis Song
- Geoffrey Irving
Citations in the corpus
- Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, background): “We also present the first application of unlikelihood training (Welleck et al., 2019a; Li et al., 2020) to the problem of de-biasing language generation (to our knowledge).”
- Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “One intriguing possibility is to apply apply likelihood and unlikelihood training to dialogue generation by rewarding correct reasoning and penalizing incorrect reasoning (Li et al., 2020).”
- Red Teaming Language Models with Language Models (2022, background): “…can address the specific biases by training the model to unlearn them, e.g., by finetuning on examples that align better with human values (Solaiman and Dennison, 2021) or reducing the likelihood of the current, offensive replies with unlikelihood training (Welleck et al., 2020; Li et al., 2020).”
- Survey of Hallucination in Natural Language Generation (2022, methodology): “Lately, the application of NLI methods [106, 169] or reinforcement learning frameworks [128] have been investigated.”
- Recipes for Building an Open-Domain Chatbot (2020, methodology): “The unlikelihood loss (Welleck et al., 2020; Li et al., 2019a) has been shown to help fix mismatches between human and model distributions across various axes, including decreasing repetitions and mitigating the issue of overrepresented vocabulary tokens.”
- Quark: Controllable Text Generation with Reinforced Unlearning (2022, background): “Complementary pre hoc methods aim to avoid learning undesired behavior at training time [78, 37, 7].”
- FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020, background): “Recently, more work has focused on semantic errors in model outputs, such as adequacy in machine translation (Tu et al., 2017), faithfulness in summarization (Cao et al., 2018), and consistency in dialogue (Li et al., 2019).”
- Memory-Based Model Editing at Scale (2022, methodology): “Because each topic contains responses of both sentiments, we make use of unlikelihood training (Li et al., 2020) in the ConvSent setting.”
- EVA2.0: Investigating Open-domain Chinese Dialogue Systems with Large-scale Pre-training (2022, background): “Although some works try to address the problem [42, 43], it is not yet entirely solved, especially in languages other than English.”
- Automatically Exposing Problems with Neural Dialog Models (2021, background): “In terms of consistency detection, Dziri et al. (2019); Welleck et al. (2019); Li et al. (2020) introduce and suggest using natural language inference to model conversation coherence.”
- ORPO: Monolithic Preference Optimization without Reference Model (2024, background): “Penalizing Undesired Generations Appending an unlikelihood penalty to the loss has demonstrated success in reducing unwanted degenerative traits in models (Welleck et al., 2019; Li et al., 2020).”
- Recent advances in deep learning based dialogue systems: a systematic survey (2021, background): “Lin et al. (2020c) introduced Levenshtein belief spans (Lev), which were short context spans relating to the user message.”
- Relating Neural Text Degeneration to Exposure Bias (2021, methodology): “Li et al. (2019) further applied unlikelihood training on dialogue tasks.”
- Entity-level Factual Consistency of Abstractive Text Summarization (2021, background): “In the dialogue generation setting, authors in (Li et al., 2019) proposed using unlikelihood to surpress logically inconsistent responses.”
- Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it? (2021, methodology): “Recently developed models that can or could be found on the leaderboards of these tasks without using ensembling are e.g. ViLBERT, LXMERT, ImageBERT and OSCAR (Lu et al., 2019; Tan and Bansal, 2019; Qi et al., 2020; Li et al., 2020b).”
- Typical Decoding for Natural Language Generation (2022, background): “, 2021) while others blame shortcomings of model architectures or training strategies (Welleck et al., 2020; Li et al., 2020).”
- How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN (2021, unknown): “These examples display a different type of inconsistency from the inconsistency observed in prior work, namely inconsistency in how to apply morphology, as opposed to factual inconsistency (Welleck et al., 2019; Li et al., 2020).”
- Linguistic calibration through metacognition: aligning dialogue agent responses with expected correctness (2020, background): “…a vanilla model directly expresses the correct level of confidence, e.g. through retraining as in Xu et al. (2020), or by training the model specifically not to output responses for which confidence and correctness don’t match through unlikelihood techniques (Welleck et al., 2019; Li et al., 2019).”
- CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization (2021, methodology): “For text generation tasks, unlikelihood training is proposed to penalize the generation of negative tokens (e.g., repeated words) and sentences (e.g., contra-dictory responses in a dialogue system) (Welleck et al., 2020; Li et al., 2020; He and Glass, 2020).”
- I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling (2020, methodology): “These include Poly-encoder (Humeau et al., 2019) based retrieval models, generative models (Roller et al., 2020), unlikelihood trained models (Li et al., 2019a), retrieve-and-refine models (Weston et al., 2018; Roller et al., 2020), models either pre-trained on a previously existing Red-dit dataset…”