Reward Shaping with Recurrent Neural Networks for Speeding up On-Line Policy Learning in Spoken Dialogue Systems

By Pei-hao Su, David Vandyke, Milica Gasic, Nikola Mrksic, Tsung-Hsien Wen, Susannah Young. 2015

52 citations recorded by the source corpus.

Statistical spoken dialogue systems have the attractive property of being able to be optimised from data via interactions with real users. However in the reinforcement learning paradigm the dialogue manager (agent) often requires significant time to explore the state-action space to learn to behave in a desirable manner. This is a critical issue when the system is trained on-line with real users where learning costs are expensive. Reward shaping is one promising technique for addressing these concerns. Here we examine three recurrent neural network (RNN) approaches for providing reward shaping information in addition to the primary (task-orientated) environmental feedback. These RNNs are trained on returns from dialogues generated by a simulated user and attempt to diffuse the overall evaluation of the dialogue back down to the turn level to guide the agent towards good behaviour faster. In both simulated and real user scenarios these RNNs are shown to increase policy learning speed. Importantly, they do not require prior knowledge of the user's goal.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Dialog-based Language Learning (2016, neutral): “Finally, dialog-based feedback could also be used as a medium to learn non-dialog based skills, e.g. natural language dialog for completing visual or physical tasks.”
RUDDER: Return Decomposition for Delayed Rewards (2018, methodology): “LSTM was already used in reinforcement learning [112] for advantage learning [4], for constructing a potential function for reward shaping by representing the return by a sum of LSTM outputs across an episode [124], and learning policies [44, 80, 45].”
Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management (2017, background): “In addition to increasing the sample-efficiency of the learning algorithms, the use of reward shaping has also been investigated in (El Asri et al., 2014; Su et al., 2015b) to enrich the reward function in order to speed up dialogue policy learning.”
Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation (2020, background): “Early studies focus on learning reward function utilizing external evaluations, e.g., user experience feedbacks (Gaˇsi´c et al., 2013), objective ratings (Su et al., 2015; Ultes et al., 2017), or a combination of multiple evaluations (Su et al., 2016; Chen et al., 2019).”
Transfer Learning in Deep Reinforcement Learning: A Survey (2020, methodology): “Examples include learning from expert demonstrations for Spoken Dialogue Systems [154], VQA [152]; or reward shaping for Sequence Generation [155], Spoken Dialogue Systems [80], QA [81], [156], and Image Caption [153], or transferring policies for Structured Prediction [157] and VQA [158].”
Learning Navigation Behaviors End-to-End With AutoRL (2018, background): “Reward shaping [21], [36] addresses this problem by introducing a proxy reward function that is less sparse than the true objective.”
On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems (2016, methodology): “The use of reward shaping has also been investigated in (El Asri et al., 2014; Su et al., 2015b) to enrich the reward function in order to speed up dialogue policy learning.”