Iterative policy learning in end-to-end trainable task-oriented neural dialog models

By Bing Liu, Ian Lane. 2017

105 citations recorded by the source corpus.

In this paper, we present a deep reinforcement learning (RL) framework for iterative dialog policy optimization in end-to-end task-oriented dialog systems. Popular approaches in learning dialog policy with RL include letting a dialog agent to learn against a user simulator. Building a reliable user simulator, however, is not trivial, often as difficult as building a good dialog agent. We address this challenge by jointly optimizing the dialog agent and the user simulator with deep RL by simulating dialogs between the two agents. We first bootstrap a basic dialog agent and a basic user simulator by learning directly from dialog corpora with supervised training. We then improve them further by letting the two agents to conduct task-oriented dialogs and iteratively optimizing their policies with deep RL. Both the dialog agent and the user simulator are designed with neural network models that can be trained end-to-end. Our experiment results show that the proposed method leads to promising improvements on task success rate and total task reward comparing to supervised training and single-agent RL training baseline models.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “Improving dialog systems with RL has largely been restricted to task-oriented dialog systems, which have a limited number of task-specific actions (Fatemi et al., 2016; Gašić et al., 2011; Liu and Lane, 2017; Liu et al., 2018; Su et al., 2017).”
Neural Approaches to Conversational AI (2018, positive): “This inspire another line of work that uses reinforcement learning to optimize end-to-end dialogue systems (Zhao and Eskénazi, 2016; Williams and Zweig, 2016; Dhingra et al., 2017; Li et al., 2017d; Braunschweiler and Papangelis, 2018; Strub et al., 2017; Liu and Lane, 2017; Liu et al., 2018a).”
Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems (2018, background): “RL by receiving feedback during users interactions showed improved model robustness against diverse dialogue scenarios (Williams and Zweig, 2016; Liu and Lane, 2017b).”
Bootstrapping a Neural Conversational Agent with Dialogue Self-Play, Crowdsourcing and On-Line Reinforcement Learning (2018, background): “Approaches that use reinforcement learning to find the optimal policy also rely on a pre-training step of supervised learning over expert dialogues in order to reduce the exploration space to make the policy learning tractable (Fatemi et al. (2016); Su et al. (2016b, 2017); Liu and Lane (2017b)).”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, background): “Policies learned in a supervised fashion exhibit great decision-making ability (Su et al., 2016; Dhingra et al., 2016; Williams et al., 2017; Liu and Lane, 2017).”
An Overview of Natural Language State Representation for Reinforcement Learning (2020, methodology): “A common strategy has been to use the hidden vector of RNNs to represent and update the environment state (Zhao & Eskenazi, 2016; Williams et al., 2017; Liu & Lane, 2017).”
Hierarchical Reinforcement Learning for Open-Domain Dialog (2019, methodology): “RL has also been used to optimize for rewards based on GAN-like discriminators trained to distinguish human-generated from model-generated text (Li et al. 2017; Yu et al. 2017).”
A Teacher-Student Framework for Maintainable Dialog Manager (2018, background): “But 3 Similar dialog model architectures can be found in recent work (Liu and Lane, 2017; Williams et al., 2017; Su et al., 2016b).”
Adversarial Learning of Task-Oriented Neural Dialog Models (2018, methodology): “We follow the design and training procedures of (Liu and Lane, 2017b) in building the model-based simulator.”
Sample Efficient Deep Reinforcement Learning for Dialogue Systems With Large Action Spaces (2018, methodology): “More recently RL learning has also been applied yielding improvements [8], [9], [10], [11].”
Sentiment Adaptive End-to-End Dialog Systems (2018, background): “Reinforcement learning (RL) is also popular in dialog system building (Zhao and Eskenazi, 2016; Liu and Lane, 2017b; Li et al., 2016).”
End-to-End Learning of Task-Oriented Dialogs (2018, methodology): “We extend the LSTM based SLU models by adding a higher level LSTM on top to model dialog context over multiple turns (Liu and Lane, 2017a).”
The Bottleneck Simulator: A Model-based Deep Reinforcement Learning Approach (2018, background): “Researchers have also investigated learning generative neural network policies operating directing on raw text through user simulations (Zhao & Eskenazi, 2016; Guo et al., 2016; Das et al., 2017; Lewis et al., 2017; Liu & Lane, 2017).”
Learning Knowledge Bases with Parameters for Task-Oriented Dialogue Systems (2020, background): “…learn a better dialogue policy via interaction (Asri et al., 2016; Li et al., 2017; Wu et al., 2019c; Peng et al., 2018), and it is especially useful in scenarios in where few or no data is available (Liu and Lane, 2017; Liu et al., 2017; Shah et al., 2018; Kreyssig et al., 2018; Li et al., 2020).”
Sequicity: Simplifying Task-oriented Dialogue Systems with Single Sequence-to-Sequence Architectures (2018, background): “Recent experiments suggest that reinforcement learning is a promising paradigm to accomplish this task (Young et al., 1 http://github.com/WING-NUS/sequicity 2013a; Cuay´ahuitl et al., 2015; Liu and Lane, 2017), when state and action spaces are carefully designed (Young et al., 2010).”