Optimizing Dialogue Management with Reinforcement Learning: Experiments with the NJFun System

By Michael Kearns, Diane J. Litman, Satinder Singh, Marilyn A. Walker. 2011

428 citations recorded by the source corpus.

Designing the dialogue policy of a spoken dialogue system involves many nontrivial choices. This paper presents a reinforcement learning approach for automatically optimizing a dialogue policy, which addresses the technical challenges in applying reinforcement learning to a working dialogue system with human users. We report on the design, construction and empirical evaluation of NJFun, an experimental spoken dialogue system that provides users with access to information about fun things to do in New Jersey. Our results show that by optimizing its performance via reinforcement learning, NJFun measurably improves system performance.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Dueling RL: Reinforcement Learning with Trajectory Preferences (2021, background): “…incorporating preference feedback in the reinforcement learning (RL) framework, which considers the problem of long-term objectives over a markov decision process (Singh et al., 2002; Ng et al., 2006; Talebi and Maillard, 2018; Ortner, 2020; Zhang and Ji, 2019; Zanette and Brunskill, 2019).”
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models (2015, methodology): “In addition to this, the features extracted from the non-goal-driven systems may be used to expand the state space representation of POMDP models (Singh et al. 2002).”
Deep Reinforcement Learning for Dialogue Generation (2016, background): “This dialogue literature thus widely applies reinforcement learning (Walker, 2000; Schatzmann et al., 2006; Gasic et al., 2013b; Singh et al., 1999; Singh et al., 2000; Singh et al., 2002) to train dialogue policies.”
Neural Approaches to Conversational AI (2018, neutral): “ere, we review a small subset of traditional approaches from the decision-theoretic view we take in this paper. Levin et al. (2000) viewed conversation as a decision making problem. Walker (2000) and Singh et al. (2002) are two early applications of reinforcement learning to manage dialogue systems.”
A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues (2016, background): “Such models are not specifically designed for the goal-oriented setting, in which dialogue systems were originally developed (Gorin, Riccardi, and Wright 1997; Young 2000; Singh et al. 2002; Young et al. 2013).”
The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems (2015, neutral): “This is in contrast to recent systems which focus on structured dialogue tasks, using a slot-filling representation [9, 26, 30].”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, background): “User simulation models are particularly useful in the development of dialogue systems based on reinforcement learning methods (Singh et al., 2002; Schatzmann et al., 2006; Pietquin and Dutoit, 2006; Frampton and Lemon, 2009; Jurčı́ček et al., 2012; Png and Pineau, 2011; Young et al., 2013).”
Generative Deep Neural Networks for Dialogue: A Short Review (2016, methodology): “Although the Seq2Seq framework is different from the well-established goal-oriented setting [Gorin et al., 1997, Young, 2000, Singh et al., 2002], these models have already been applied to several real-world applications, with Microsoft’s system Xiaoice [Markoff and Mozur, 2015] and Google’s Smart…”
Deal or No Deal? End-to-End Learning of Negotiation Dialogues (2017, methodology): “RL has been widely used to improve dialogue managers, which manage transitions between dialogue states (Singh et al., 2002; Pietquin et al., 2011; Rieser and Lemon, 2011; Gaˇsic et al., 2013; Fatemi et al., 2016).”
A Survey of Reinforcement Learning Informed by Natural Language (2019, methodology): “Since their inception, RL algorithms have been successful in applications such as continuous control [White and Sofge, 1992], dialogue systems [Singh et al. , 2002], and board games [ Tesauro, 1995 ] .”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “Although RL had some successes in the past [31], [53], [74], [81], previous approaches lacked scalability and were inherently limited to fairly low-dimensional problems.”
Hybrid Reinforcement/Supervised Learning of Dialogue Policies from Fixed Data Sets (2008, methodology): “Because most previous work has only differentiated between a small number of possible dialogue contexts, they have been able to perform these estimates for each state independently (e.g., Singh et al. 2002; Pietquin 2004).”
Controlling User Perceptions of Linguistic Style: Trainable Generation of Personality Traits (2011, methodology): “the parameterization of a dialogue manager and a TTS component, in order to project a consistent personality to the user, or to use a reinforcement learning approach to train the dialogue manager (Walker, Fromer, and Narayanan 1998; Singh et al. 2002), with personality judgments as the objective fun”
Natural Language Generation as Planning Under Uncertainty for Spoken Dialogue Systems (2009, methodology): “The first data set, see (Stent et al., 2002), comprises 1024 ratings by 16 subjects (where we only use the speech-based half, n = 512) on the following presentation strategies: RECOMMEND, COMPARE, SUMMARY.”
Frames: a corpus for adding memory to goal-oriented dialogue systems (2017, background): “Goal-oriented, information-retrieving dialogue systems have been designed traditionally to help users find items in a database given a set of constraints (Singh et al., 2002; Raux et al., 2003; El Asri et al., 2014; Laroche et al., 2011).”
Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning (2017, background): “…a pipeline, typically including language understanding, dialog state tracking, action selection policy, and language generation (Levin et al., 2000; Singh et al., 2002; Williams and Young, 2007; Williams, 2008; Hori et al., 2009; Lee et al., 2009; Griol et al., 2008; Young et al., 2013; Li et…”
Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning (2016, background): “Reinforcement Learning (RL): RL has been a popular approach for learning the optimal dialog policy of a task-oriented dialog system (Singh et al., 2002; Williams and Young, 2007; Georgila and Traum, 2011; Lee and Eskenazi, 2012).”
I love your chain mail! Making knights smile in a fantasy game world: Open-domain goal-oriented dialogue agents (2019, background): “Typically, they used RL to improve dialogue managers, which manage transitions between dialogue states (Singh et al., 2002; Pietquin et al., 2011; Rieser & Lemon, 2011; Gasic et al., 2013; Fatemi et al., 2016).”