Continuously Learning Neural Dialogue Management

By Pei-hao Su, Milica Gasic, Nikola Mrksic, Lina Maria Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, Susannah Young. 2016

123 citations recorded by the source corpus.

We describe a two-step approach for dialogue management in task-oriented spoken dialogue systems. A unified neural network framework is proposed to enable the system to first learn by supervision from a set of dialogue data and then continuously improve its behaviour via reinforcement learning, all using gradient-based algorithms on one single model. The experiments demonstrate the supervised model's effectiveness in the corpus-based evaluation, with user simulation, and with paid human subjects. The use of reinforcement learning further improves the model's performance in both interactive settings, especially under higher-noise conditions.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Learning Through Dialogue Interactions (2016, background): “It also include a collection of goal-oriented dialogue systems (Wen et al., 2016; Su et al., 2016; Bordes & Weston, 2016), which complete a certain task such as booking a ticket or making a reservation at a restaurant.”
Adversarial Learning for Neural Dialogue Generation (2017, background): “For example, Su et al. (2016) combine reinforcement learning with neural generation on tasks with real users.”
Deep Reinforcement Learning for Dialogue Generation (2016, background): “Neural response generation (Sordoni et al., 2015; Shang et al., 2015; Vinyals and Le, 2015; Li et al., 2016a; Wen et al., 2015; Yao et al., 2015; Luan et al., 2016; Xu et al., 2016; Wen et al., 2016; Li et al., 2016b; Su et al., 2016) is of growing interest.”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, background): “Researchers have also proposed training dialogue systems online through live interaction with humans, and offline using user simulator models and reinforcement learning methods (Levin et al., 1997; Georgila et al., 2006; Paek, 2006; Schatzmann et al., 2007; Jung et al., 2009; Schatzmann and Young, 2”
Dialogue Learning With Human-In-The-Loop (2016, neutral): “Efforts include Markov Decision Processes (MDPs) (Levin et al., 1997; 2000; Walker et al., 2003; Pierac-cini et al., 2009), POMDP models (Young et al., 2010; 2013; Gaˇsic et al., 2013; 2014) and policy learning (Su et al., 2016).”
Dynamic Dialogue Policy for Continual Reinforcement Learning (2022, background): “The approaches directly vectorise the belief state, hence the size of the input vector depends on the domain (as different domains for instance have different numbers of slots) (Su et al., 2016; Lipton et al., 2018; Weisz et al., 2018; Takanobu et al., 2019; Zhu et al., 2020).”
End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient (2017, methodology): “Usually, these algorithms are followed by training with reinforcement learning to further refine the model [21, 28, 23, 27].”
Bootstrapping a Neural Conversational Agent with Dialogue Self-Play, Crowdsourcing and On-Line Reinforcement Learning (2018, methodology): “Approaches that use reinforcement learning to ﬁnd the optimal policy also rely on a pre-training step of supervised learning over expert dialogues in order to reduce the exploration space to make the policy learning tractable (Fatemi et al. (2016); Su et al. (2016b, 2017); Liu and Lane (2017b)).”
Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management (2017, methodology): “A number of authors have proposed training a standard neural-network policy in two stages (Fatemi et al., 2016; Su et al., 2016a; Williams et al., 2017).”
Two are Better than One: An Ensemble of Retrieval- and Generation-Based Dialog Systems (2016, background): “In the above scenarios, the dialogs are domain-speciﬁc, and a typical approach to such in-domain systems is by human engineering, for example, using manually constructed ontologies [23], natural language templates [20], and even predeﬁned dialog states [25].”
Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning (2017, background): “Speciﬁ-cally related to HCNs, past work has implemented the policy as feed-forward neural networks (Wen et al., 2016), trained with supervised learning followed by reinforcement learning (Su et al., 2016).”
Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings (2017, background): “Despite their success (Wen et al., 2017; Dhingra et al., 2017; Su et al., 2016), the requirement for handcrafted slots limits their scalability to new domains and burdens data collection with extra state labeling.”
Under Review as a Conference Paper at Iclr 2017 Learning through Dialogue Interactions (2016, background): “It also include a collection of goal-oriented dialogue systems (Wen et al., 2016; Su et al., 2016; Bordes & Weston, 2016), which complete a certain task such as booking a ticket or making a reservation at a restaurant.”
Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning (2018, background): “…systems that use this action space representation include conversational agents such as personal assistants (Dhingra et al., 2016; Li et al., 2017; Su et al., 2016; Lipton et al., 2016b; Liu et al., 2017; Zhao and Eskenazi, 2016; Wu et al., 2016), travel planners (Peng et al., 2017),…”
A Teacher-Student Framework for Maintainable Dialog Manager (2018, methodology): “…(Mnih et al., 2013, 2015) has been applied to optimize the dialog manager in an “end-to-end” way, including deep Q-Network (Lipton et al., 2017; Li et al., 2017b; Peng et al., 2017; Zhao and Eskenazi, 2016) and policy gradient methods (Williams et al., 2017; Su et al., 2016b; Dhingra et al., 2017).”
The Bottleneck Simulator: A Model-based Deep Reinforcement Learning Approach (2018, unknown): “…and actions (Eckert et al., 1997; Levin et al., 2000; Chung, 2004; Cuay´ahuitl et al., 2005; Georgila et al., 2006; Schatzmann et al., 2007; Heeman, 2009; Traum et al., 2008; Lee & Eskenazi, 2012; Khouzaimi et al., 2017; L´opez-C´ozar, 2016; Su et al., 2016; Fatemi et al., 2016; Asri et al., 2016).”