Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning

By Tiancheng Zhao, Maxine Eskénazi. 2016

277 citations recorded by the source corpus.

This paper presents an end-to-end framework for task-oriented dialog systems using a variant of Deep Recurrent Q-Networks (DRQN). The model is able to interface with a relational database and jointly learn policies for both language understanding and dialog strategy. Moreover, we propose a hybrid algorithm that combines the strength of reinforcement learning and supervised learning to achieve faster learning speed. We evaluated the proposed model on a 20 Question Game conversational game simulator. Results show that the proposed method outperforms the modular-based baseline and learns a distributed representation of the latent dialog state.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Keep CALM and Explore: Language Models for Action Generation in Text-based Games (2020, background): “More broadly, the concept of generating candidates and re-ranking has been studied in other interactive lanugage tasks such as dialogue (Zhao and Eskenazi, 2016; Williams et al., 2017; Song et al., 2016; Chen et al., 2017) and communication games (Lazaridou et al., 2020).”
Neural Approaches to Conversational AI (2018, positive): “This inspire another line of work that uses reinforcement learning to optimize end-to-end dialogue systems (Zhao and Eskénazi, 2016; Williams and Zweig, 2016; Dhingra et al., 2017; Li et al., 2017d; Braunschweiler and Papangelis, 2018; Strub et al., 2017; Liu and Lane, 2017; Liu et al., 2018a).”
Incremental Learning from Scratch for Task-Oriented Dialogue Systems (2019, methodology): “Data-driven methods, such as reinforcement learning (Williams et al., 2017; Zhao and Eskenazi, 2016; Li et al., 2017) and supervised learning (Wen et al.”
MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling (2018, background): “, 2016), and even end-to-end dialogue modelling (Zhao and Eskenazi, 2016; Wen et al., 2017; Eric et al., 2017).”
Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems (2018, background): “2017; Liu and Lane, 2017a) and deep reinforcement learning (RL) based systems (Zhao and Eskenazi, 2016; Li et al., 2017; Peng et al., 2017) have been studied in the literature.”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, background): “ly defines a set of user intents, and for each intent defines a set of dialog acts, slot-value pairs. 3Dialog management performs both dialog state tracking [47, 89] and response selection via policy [71, 99, 132, 183]. 4Recently, there are end-to-end methods [9, 112, 180] that output a response giv”
Iterative policy learning in end-to-end trainable task-oriented neural dialog models (2017, background): “User simulators used in most of the recent RL based dialog models [19, 23, 20] are designed with expert knowledge and complex rules.”
A Survey on Dialogue Systems: Recent Advances and New Frontiers (2017, background): “[107] first presented an end-to-end reinforcement learning approach to jointly train dialogue state tracking and policy learning in the dialogue management in order to optimize the system actions more robustly.”
Sentiment Adaptive End-to-End Dialog Systems (2018, background): “Reinforcement learning (RL) is also popular in dialog system building (Zhao and Eskenazi, 2016; Liu and Lane, 2017b; Li et al., 2016).”
Learning Robust Dialog Policies in Noisy Environments (2017, background): “Zhao and Eskenazi [36] similarly used a deep recurrent Q-network (DRQN) to learn both state tracking and dialog policy for task-oriented dialogs.”
Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders (2017, methodology): “example, dialog acts (Poesio and Traum, 1998) have been widely used in the dialog managers (Litman and Allen, 1987; Raux et al., 2005; Zhao and Eskenazi, 2016) to represent the propositional function of the system.”
Beyond task success: A closer look at jointly learning to see, ask, and GuessWhat (2018, unknown): “Recent works combine conventional symbolic with neural approaches (Williams et al., 2017; Zhao and Eskenazi, 2016; Rastogi et al., 2018), but all focus on language-only dialogue.”
A Teacher-Student Framework for Maintainable Dialog Manager (2018, methodology): “…(Mnih et al., 2013, 2015) has been applied to optimize the dialog manager in an “end-to-end” way, including deep Q-Network (Lipton et al., 2017; Li et al., 2017b; Peng et al., 2017; Zhao and Eskenazi, 2016) and policy gradient methods (Williams et al., 2017; Su et al., 2016b; Dhingra et al., 2017).”
Conversational Recommender System (2018, methodology): “[32] was among the first works of building an end-to-end dialogue system.”
Sample Efficient Deep Reinforcement Learning for Dialogue Systems With Large Action Spaces (2018, methodology): “More recently RL learning has also been applied yielding improvements [8]–[12].”
SOLOIST: Few-shot Task-Oriented Dialog with A Single Pre-trained Auto-regressive Model (2020, background): “End-to-end trainable dialog systems have been studied in (Wen et al., 2016; Zhao and Eskenazi, 2016; Li et al., 2017; Williams et al., 2017; Lei et al., 2018; Gao et al., 2019; Zhang et al., 2019a).”
End-to-End Reinforcement Learning of Dialogue Agents for Information Access (2016, methodology): “Previous end-to-end systems achieved this by con-a structing a symbolic query from the current belief states of the agent and retrieving results from the database which match the query (Wen et al., 2016b; Williams and Zweig, 2016; Zhao and Eskenazi, 2016).”
Task-Oriented Dialogue as Dataflow Synthesis (2020, background): “(2019) report greater success on a generation-focused task, and promising results have also been obtained from hybrid neuro-symbolic dialogue systems (Zhao and Eskenazi, 2016; Williams et al., 2017; Wen et al., 2017; Gao et al., 2019).”
Adversarial Language Games for Advanced Natural Language Intelligence (2019, background): “, booking hotels or restaurants) (Goddeau et al., 1996; Williams et al., 2013; Henderson et al., 2014; Cuayáhuitl et al., 2015; Zhao and Eskenazi, 2016), while none-goal-oriented dialogue systems (also known as chatbots) interact”