Policy Networks with Two-Stage Training for Dialogue Systems

By Mehdi Fatemi, Layla El Asri, Hannes Schulz, Jing He, Kaheer Suleman. 2016

110 citations recorded by the source corpus.

In this paper, we propose to use deep policy networks which are trained with an advantage actor-critic method for statistically optimised dialogue systems. First, we show that, on summary state and action spaces, deep Reinforcement Learning (RL) outperforms Gaussian Processes methods. Summary state and action spaces lead to good performance but require pre-engineering effort, RL knowledge, and domain expertise. In order to remove the need to define such summary spaces, we show that deep RL can also be trained efficiently on the original state and action spaces. Dialogue systems based on partially observable Markov decision processes are known to require many dialogues to train, which makes them unappealing for practical deployment. We show that a deep RL method based on an actor-critic architecture can exploit a small amount of data very efficiently. Indeed, with only a few hundred dialogues collected with a handcrafted policy, the actor-critic deep learner is considerably bootstrapped from a combination of supervised and batch RL. In addition, convergence to an optimal policy is significantly sped up compared to other deep RL methods initialized on the data with batch RL. All experiments are performed on a restaurant domain derived from the Dialogue State Tracking Challenge 2 (DSTC2) dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “Improving dialog systems with RL has largely been restricted to task-oriented dialog systems, which have a limited number of task-specific actions (Fatemi et al., 2016; Gašić et al., 2011; Liu and Lane, 2017; Liu et al., 2018; Su et al., 2017).”
Surprising Negative Results for Generative Adversarial Tree Search (2018, background): “For example (Cuayáhuitl, 2016; Fatemi et al., 2016; Wen et al., 2016) investigate DRL for dialogue policy learning, with (Lipton et al., 2018) addressing the efﬁciency of exploration.”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, background): “Researchers have also proposed training dialogue systems online through live interaction with humans, and offline using user simulator models and reinforcement learning methods (Levin et al., 1997; Georgila et al., 2006; Paek, 2006; Schatzmann et al., 2007; Jung et al., 2009; Schatzmann and Young, 2”
Deal or No Deal? End-to-End Learning of Negotiation Dialogues (2017, methodology): “RL has been widely used to improve dialogue managers, which manage transitions between dialogue states (Singh et al., 2002; Pietquin et al., 2011; Rieser and Lemon, 2011; Gaˇsic et al., 2013; Fatemi et al., 2016).”
BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems (2016, methodology): “2016) and RL for learning dialogue policies (Levin, Pieraccini, and Eckert 2000; Singh et al. 2000; Williams and Young 2007; Gašić et al. 2010; Fatemi et al. 2016).”
Frames: a corpus for adding memory to goal-oriented dialogue systems (2017, background): “Performing more than one action per turn is a challenge when using reinforcement learning (Pietquin et al., 2011; Gašić et al., 2012; Fatemi et al., 2016) and, to our knowledge, has only been tackled in a simulated setting (Laroche et al.”
Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management (2017, methodology): “Note that GPRL was used by Fatemi et al. (2016) to compare with deep RL but no uncertainty estimate was used to guide exploration and as a result had relatively poor performance.”
Efﬁcient Exploration for Dialogue Policy Learning with BBQ Networks & Replay Buffer Spiking (2016, methodology): “Thus, reinforcement learning (RL), in which policies are learned through interaction with an unknown and possibly changing environment, has emerged as a popular alternative [Singh et al., 2000, Gaši´c et al., 2010, Fatemi et al., 2016].”
MAKING KNIGHTS SMILE IN A FANTASY GAME WORLD (2019, background): “Typically, they used RL to improve dialogue managers, which manage transitions between dialogue states (Singh et al., 2002; Pietquin et al., 2011; Rieser & Lemon, 2011; Gasic et al., 2013; Fatemi et al., 2016).”
Combating Deep Reinforcement Learning's Sisyphean Curse with Intrinsic Fear (2016, background): “[2016], dialogue systems Fatemi et al. [2016], Lipton et al.”
Interactive reinforcement learning for task-oriented dialogue management (2016, methodology): “We follow a normalized reward scheme similar to Fatemi et al. (2016): the model receives a reward of +1 for successful dialogues and -1 for unsuccessful dialogues, with a penalty of -0.03 for every turn, to encourage shorter dialogues.”
Efficient Exploration for Dialog Policy Learning with Deep BBQ Networks \& Replay Buffer Spiking (2016, methodology): “While we don’t compare against Gaussian processes, we point to the recent work of Fatemi et al. [2016], which compares deep reinforcement learning methods (both policy gradient and Q-learning approaches) to GP-SARSA [Engel et al., 2005] on a similar dialog policy problem.”
Distributed Structured Actor-Critic Reinforcement Learning for Universal Dialogue Management (2020, background): “are investigated for dialogue policy optimisation, including A2C [20], [21], eNAC [2], and ACER [22].”
Sample-Efficient Deep RL with Generative Adversarial Tree Search (2018, background): “For example (Cuayáhuitl, 2016; Fatemi et al., 2016; Wen et al., 2016) investigate DRL for dialogue policy learning, with (Lipton et al.”
Deep Reinforcement Learning: An Overview (2017, unknown): “A move is then selected.”
An Overview of Natural Language State Representation for Reinforcement Learning (2020, methodology): “Subsequent works followed the same approach (Ansari et al., 2018; Yuan et al., 2018; Jain et al., 2020) or used variants of continuous vectors built by neural networks (He et al., 2016; Foerster et al., 2016).”