Learning from real users: rating dialogue success with neural networks for reinforcement learning in spoken dialogue systems

By Pei-hao Su, David Vandyke, Milica Gasic, Dongho Kim, Nikola Mrksic, Tsung-Hsien Wen, Susannah Young. 2015

70 citations recorded by the source corpus.

To train a statistical spoken dialogue system (SDS) it is essential that an accurate method for measuring task success is available. To date training has relied on presenting a task to either simulated or paid users and inferring the dialogue's success by observing whether this presented task was achieved or not. Our aim however is to be able to learn from real users acting under their own volition, in which case it is non-trivial to rate the success as any prior knowledge of the task is simply unavailable. User feedback may be utilised but has been found to be inconsistent. Hence, here we present two neural network models that evaluate a sequence of turn-level features to rate the success of a dialogue. Importantly these models make no use of any prior knowledge of the user's task. The models are trained on dialogues generated by a simulated user and the best model is then used to train a policy on-line which is shown to perform at least as well as a baseline system using prior knowledge of the user's task. We note that the models should also be of interest for evaluating SDS and for monitoring a dialogue in rule-based SDS.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Continuously Learning Neural Dialogue Management (2016, background): “In this framework, the system learns by a trial and error process governed by a potentially delayed learning objective, a reward function , that determines dialogue success (El Asri et al., 2014; Su et al., 2015; Vandyke et al., 2015; Su et al., 2016).”
Bootstrapping a Neural Conversational Agent with Dialogue Self-Play, Crowdsourcing and On-Line Reinforcement Learning (2018, background): “This can be mitigated by continuously improving the deployed agent from interactions with actual users via on-line learning (Gašić et al. (2011); Su et al. (2015, 2016a)).”
Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management (2017, background): “Typically it provides a positive reward for success plus a per turn penalty to encourage short dialogues (El Asri et al., 2014; Su et al., 2015a; Vandyke et al., 2015; Su et al., 2016b).”
A Survey on Dialogue Systems: Recent Advances and New Frontiers (2017, methodology): “[70] proposed to use two neural network models to evaluate a sequence of turn-level features to rate the success of a dialogue.”
On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems (2016, background): “From the figure we can clearly see the colour gradient from the top left (shorter dialogues) to the bottom right (longer dialogues) for the positive Subj labels.”
Latent Intention Dialogue Models (2017, methodology): “We then evaluated our model on task success rate (Su et al., 2015) and BLEU score (Papineni et al., 2002) as in Wen et al (2016; 2017) in which the model is used to predict each system response in the held-out test set.”
Adversarial Learning of Task-Oriented Neural Dialog Models (2018, background): “Su et al. (2015) studied us-ing convolutional neural networks in rating dialog success.”
A Network-based End-to-End Trainable Task-oriented Dialogue System (2016, background): “, 2002), entity matching rate and objective task success rate (Su et al., 2015).”
Generative Deep Neural Networks for Dialogue: A Short Review (2016, background): “…investigated applying neural networks to the different components of a standard dialogue system, including natural language understanding, natural language generation, dialogue state tracking and evaluation [Wen et al., 2016, 2015, Henderson et al., 2013, Mrkši´c et al., 2015, Su et al., 2015].”
Learning End-to-End Goal-Oriented Dialog (2016, methodology): “The data for all tasks is available from http://fb.ai/babi .”
Conditional Generation and Snapshot Learning in Neural Dialogue Systems (2016, methodology): “Three evaluation metrics were used: BLEU score (on top-1 and top-5 candidates) (Papineni et al., 2002), slot matching rate and objective task success rate (Su et al., 2015).”
Recent Trends in Deep Learning Based Natural Language Processing [Review Article] (2018, background): “Secondly, the need for training the reward functions makes such models hard to design and measure at run time [112], [113].”
Policy committee for adaptation in multi-domain spoken dialogue systems (2015, methodology): “At the end of each dialogue, a recurrent neural network (RNN) model was used to predict the dialogue success used as the reinforcement feedback [21].”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, methodology): “Vijayakumar et al. (2016) combined a dissimilarity term into the beam search objective and proposed Diverse Beam Search (DBS) to promote diversity.”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, background): “…Gašić et al., 2012; Su et al., 2013; Gasic et al., 2013; Pietquin and Hastie, 2013; Young et al., 2013; Mohan and Laird, 2014; Su et al., 2015; Piot et al., 2015; Cuayáhuitl et al., 2015; Hiraoka et al., 2016; Fatemi et al., 2016; Asri et al., 2016; Williams and Zweig, 2016; Su et al., 2016).”