On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems

By Pei-hao Su, Milica Gasic, Nikola Mrksic, Lina Maria Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, Susannah Young. 2016

174 citations recorded by the source corpus.

The ability to compute an accurate reward function is essential for optimising a dialogue policy via reinforcement learning. In real-world applications, using explicit user feedback as the reward signal is often unreliable and costly to collect. This problem can be mitigated if the user's intent is known in advance or data is available to pre-train a task success predictor off-line. In practice neither of these apply for most real world applications. Here we propose an on-line learning framework whereby the dialogue policy is jointly trained alongside the reward model via active learning with a Gaussian process model. This Gaussian process operates on a continuous space dialogue representation generated in an unsupervised fashion using a recurrent neural network encoder-decoder. The experimental results demonstrate that the proposed framework is able to significantly reduce data annotation costs and mitigate noisy user feedback in dialogue policy learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Recent Trends in Deep Learning Based Natural Language Processing (2017, background): “Secondly, the need for training the reward functions makes such models hard to design and measure at run time [116, 117].”
End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient (2017, background): “Recently, many RL-based algorithms have been proposed and achieved good results [15, 5, 30, 22].”
Continuously Learning Neural Dialogue Management (2016, methodology): “In this framework, the system learns by a trial and error process governed by a potentially delayed learning objective, a reward function , that determines dialogue success (El Asri et al., 2014; Su et al., 2015; Vandyke et al., 2015; Su et al., 2016).”
Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems (2018, background): “…components for natural language understanding (NLU) (Mesnil et al., 2015; Liu and Lane, 2016; Hakkani-T¨ur et al., 2016), dialogue state tracking (DST) (Henderson et al., 2014c; Mrkˇsi´c et al., 2016), and dialogue policy learning (Gasic and Young, 2014; Shah et al., 2016; Su et al., 2016, 2017).”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, background): “3Dialog management performs both dialog state tracking [47, 89] and response selection via policy [71, 99, 132, 183].”
Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management (2017, methodology): “Typically it provides a positive reward for success plus a per turn penalty to encourage short dialogues (El Asri et al., 2014; Su et al., 2015a; Vandyke et al., 2015; Su et al., 2016b).”
Global-to-local Memory Pointer Networks for Task-Oriented Dialogue (2019, methodology): “, 2018), dialogue management (Su et al., 2016), and natural language generation (Sharma et al.”
Iterative policy learning in end-to-end trainable task-oriented neural dialog models (2017, background): “Conventional task-oriented dialog systems are designed with complex pipelines [4, 5, 6, 7], and there are usually separately developed modules for natural language understanding (NLU) [8, 9, 10, 11], dialog state tracking (DST) [12, 13], and dialog management (DM) [14, 15, 16].”
Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog (2019, background): “, 2014), online active learning from user feedback (Su et al., 2016), multiobject RL to aggregate measurements of various aspects of user satisfaction (Ultes et al.”
Adversarial Learning of Task-Oriented Neural Dialog Models (2018, methodology): “Su et al. (2016) further proposed an online active learning method based on Gaussian process for dialog reward learning.”
A Teacher-Student Framework for Maintainable Dialog Manager (2018, methodology): “Recently, reinforcement learning approaches have been applied to optimize dialog systems through interaction with a user simulator or employed real users online (Gaˇsi´c et al., 2011; Su et al., 2016a; Li et al., 2016, Figure 1: An example of a task-oriented dialog after the system comes online.”
A Theoretical Framework for Conversational Search (2017, background): “[32] describe a more sophisticated restaurant recommendation system, where queries are matched to a complete item, which can then be refined or further metadata can be requested.”
Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation (2020, background): “Early studies focus on learning reward function utilizing external evaluations, e.g., user experience feedbacks (Gaˇsi´c et al., 2013), objective ratings (Su et al., 2015; Ultes et al., 2017), or a combination of multiple evaluations (Su et al., 2016; Chen et al., 2019).”
A Network-based End-to-End Trainable Task-oriented Dialogue System (2016, background): “Also, the reward functions needed to train such models are difficult to design and hard to measure at run-time (Su et al., 2015; Su et al., 2016).”
Building a Conversational Agent Overnight with Dialogue Self-Play (2018, background): “Finally, in contrast to a rule-based bot, a neural dialogue agent is amenable to further improvement from direct user interactions via reinforcement learning (Su et al. (2016); Liu et al. (2017)), opening up the possibility of life-long improvement in the quality of the dialogue agent.”
CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases (2019, background): “…Walker et al., 2002; Raux et al., 2005; Mrksic et al., 2015; Asri et al., 2017; Budzianowski et al., 2018) and modelling innovation (Artzi and Zettlemoyer, 2011; Henderson et al., 2013; Lee and Dernoncourt, 2016; hao Su et al., 2016; Dhingra et al., 2016; Li et al., 2016; Mrksic et al., 2017).”
Interactive reinforcement learning for task-oriented dialogue management (2016, methodology): “Fatemi et al. (2016); Su et al. (2016a) describe a two-stage training method which also alleviates the need for summary spaces or restricted action sets, by teaching the network to identify good actions in each state by first training on a corpus of Wizard of Oz (WOz) dialogues, where human agents…”