Deep Reinforcement Learning for Dialogue Generation

By Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, Jianfeng Gao. 2016

1,424 citations recorded by the source corpus.

Recent neural models of dialogue generation offer great promise for generating responses for conversational agents, but tend to be shortsighted, predicting utterances one at a time while ignoring their influence on future outcomes. Modeling the future direction of a dialogue is crucial to generating coherent, interesting dialogues, a need which led traditional NLP models of dialogue to draw on reinforcement learning. In this paper, we show how to integrate these goals, applying deep reinforcement learning to model future reward in chatbot dialogue. The model simulates dialogues between two virtual agents, using policy gradient methods to reward sequences that display three useful conversational properties: informativity (non-repetitive turns), coherence, and ease of answering (related to forward-looking function). We evaluate our model on diversity, length as well as with human judges, showing that the proposed algorithm generates more interactive responses and manages to foster a more sustained conversation in dialogue simulation. This work marks a first step towards learning a neural conversational model based on the long-term success of dialogues.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

The Curious Case of Neural Text Degeneration (2019, methodology): “(2) Computing the optimum argmax sequence from recurrent neural language models is not tractable, so consider two prominent decoding methods for approximating the argmax: Beam search is the most commonly used approximation in practice (Li et al., 2016c; Shen et al., 2017; Wiseman et al., 2017).”
QuAC: Question Answering in Context (2018, background): “19 Dialog ﬁts into an increasing interest in open domain dialog, mostly studied in the context of social chit-chat (Li et al., 2016; Ritter et al., 2011; Fang et al., 2017; Ghazvininejad et al., 2018).”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, methodology): “Selfchats have been used for evaluating model quality (Li et al., 2016, 2019; Ghandeharioun et al., 2019) demonstrating that they are a useful proxy for human-rated quality metrics.”
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons (2019, methodology): “The two most used approaches, single-turn pairwise evaluation (Vinyals and Le 2015; Li et al. 2016b), and multi-turn Likert scores (Venkatesh et al. 2017; Zhang et al. 2018; See et al. 2019; Dinan et al. 2019b; Dinan et al. 2019a) have serious limitations.”
Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, methodology): “the learning process.Yao et al.(2016) use the IDF value of generated sentences as a reward signal.Xing et al. (2017) use topics as an additional input while decoding to produce more speciﬁc responses.Li et al. (2016b) add personal information to make system responses more user speciﬁc.Li et al.(2017”
Towards Coherent and Cohesive Long-form Text Generation (2018, background): “Only recently have there been efforts to provide more relevant objectives, such as consistency and repetition in a text (Li et al., 2015, 2016a; Holtzman et al., 2018).”
Ethical Challenges in Data-Driven Dialogue Systems (2017, background): “Modern training of complex dialogue systems often requires specialized training schedules (Li et al. 2016) that are difﬁcult to replicate.”
Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “This includes REINFORCE (Williams, 1992) for Machine Translation (Ranzato et al., 2016), actor critic for Abstractive Summarization (Paulus et al., 2018), Image-to-Text (Liu et al., 2016), Dialogue Generation (Li et al., 2016b), and Video Captioning (Pasunuru & Bansal, 2017).”
Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, background): “…(Wu et al., 2016; Nguyen et al., 2017; Kiegeland & Kreutzer, 2021), summarization (Stiennon et al., 2020; Paulus et al., 2017), dialogue (Li et al., 2016; Zhou et al., 2017; Jaques et al., 2020), image captioning (Rennie et al., 2017), question generation (Pang & He, 2021), text-games…”
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, background): “Distinct-n score, a measure of text diversity in terms of the frequency of repetitions within a single sample x, proposed by (Li et al., 2016a).”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “RL in the open-domain dialog setting is less explored (Li et al., 2016, 2017b, 2018).”
Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control (2016, methodology): “This is accomplished by maintaining a ﬁxed copy of a sequence generation RNN pre-trained on data, which is termed the Reward RNN .”
Visual Dialog (2016, background): “While some of the earliest developed chatbots were rule-based [56], end-to-end learning based approaches are now being actively explored [7,9,19,24,45,46,53].”
Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization (2018, background): “Adversarial learning [6, 26] has been shown to be successful in dialog generation, translation, image captioning and a series of natural language generation tasks [7, 12, 27, 28, 29, 30, 31, 32, 33, 34].”
Personalizing Dialogue Agents: I have a dog, do you have pets too? (2018, methodology): “For the chit-chat setting, the most relevant work is (Li et al., 2016a).”
Adversarial Learning for Neural Dialogue Generation (2017, methodology): “, 2016a), repetitive, and short-sighted (Li et al., 2016d).”
Learning to Write with Cooperative Discriminators (2018, background): “For open-ended generation tasks such as our own, human evaluation has been found to be the only reliable measure (Li et al., 2016b; Wiseman et al., 2017).”