Hybrid Reinforcement/Supervised Learning of Dialogue Policies from Fixed Data Sets
By James Henderson, Oliver Lemon, Kallirroi Georgila. 2008
159 citations recorded by the source corpus.
Abstract We propose a method for learning dialogue management policies from a fixed data set. The method addresses the challenges posed by Information State Update (ISU)-based dialogue systems, which represent the state of a dialogue as a large set of features, resulting in a very large state space and a huge policy space. To address the problem that any fixed data set will only provide information about small portions of these state and policy spaces, we propose a hybrid model that combines reinforcement learning with supervised learning. The reinforcement learning is used to optimize a measure of dialogue reward, while the supervised learning is used to restrict the learned policy to the portions of these spaces for which we have data. We also use linear function approximation to address the need to generalize from a fixed amount of data to large state spaces. To demonstrate the effectiveness of this method on this challenging task, we trained this model on the COMMUNICATOR corpus, to which we have added annotations for user actions and Information States. When tested with a user simulation trained on a different part of the same data set, our hybrid model outperforms a pure supervised learning model and a pure reinforcement learning model. It also outperforms the hand-crafted systems on the COMMUNICATOR data, according to automatic evaluation measures, improving over the average COMMUNICATOR system policy by 10%. The proposed method will improve techniques for bootstrapping and automatic optimization of dialogue management policies from limited initial data sets.
Subfields: Artificial Intelligence, Reinforcement Learning, Dialogue & Conversational AI
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Jianfeng Gao
- Yoshua Bengio
- Bing Liu
- Richard Socher
- Mohit Bansal
- Percy Liang
- Caiming Xiong
In Reinforcement Learning
- Sergey Levine
- Ofir Nachum
- Aviral Kumar
- George Tucker
- Pieter Abbeel
- Shixiang Shane Gu
- Chelsea Finn
- Justin Fu
In Dialogue & Conversational AI
- Milica Gasic
- Pei-hao Su
- Susannah Young
- Blaise Thomson
- Jason Williams
- Stefan Ultes
- Paweł Budzianowski
- Bing Liu
Citations in the corpus
- Human-centric Dialog Training via Offline Reinforcement Learning (2020, neutral): “Henderson et al. (2008) combine using reinforcement learning to optimize dialog reward with using supervised learning to restrict the conversation to be close to the training data.”
- Neural Approaches to Conversational AI (2018, positive): “Learning a good policy from scratch often requires many data, but the process can be significantly sped up by restricting the policy search using expert-generated dialogues (Henderson et al., 2008) or teacher advice (Chen et al.”
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning (2020, methodology): “Offline reinforcement learning using large datasets has also been used in real-world systems where evaluation using a simulator is not possible, such as in robotics [Cabi et al., 2019] and dialogue systems [Henderson et al., 2008, Pietquin et al., 2011, Jaques et al., 2019].”
- RL Unplugged: Benchmarks for Offline Reinforcement Learning (2020, background): “…reinforcement learning to difficult real-world domains such as robots (Cabi et al., 2019; Gu et al., 2017; Kalashnikov et al., 2018) or dialog (Henderson et al., 2008; Jaques et al., 2019; Pietquin et al., 2011), but it is often difficult to do thorough evaluations in these domains for the…”
- Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems (2018, methodology): “Recent works (Henderson et al., 2008; Williams et al., 2017; Liu et al., 2017) explored pre-training the dialogue model using human-human or human-machine dialogue”
- Natural Language Generation as Planning Under Uncertainty for Spoken Dialogue Systems (2009, methodology): “…previous work we treat NLG as a statistical sequential planning problem, analogously to current statistical approaches to Dialogue Management (DM), e.g. (Singh et al., 2002; Henderson et al., 2008; Rieser and Lemon, 2008a) and “conversation as action under uncertainty” (Paek and Horvitz, 2000).”
- Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management (2017, methodology): “Henderson et al. (2008) proposed a hybrid SL/RL model that, in order to ensure tractability in policy optimisation, performed exploration only on the states in a dialogue corpus.”
- Iterative policy learning in end-to-end trainable task-oriented neural dialog models (2017, methodology): “Our system is trained by a combination of SL and deep RL methods, as it is shown that RL training may effectively improved the system robustness and dialog success rate [29, 20, 30].”
- RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning (2020, background): “…reinforcement learning to difficult real-world domains such as robots [Cabi et al., 2019, Gu et al., 2017, Kalashnikov et al., 2018] or dialog [Henderson et al., 2008, Pietquin et al., 2011, Jaques et al., 2019], but it is often difficult to do thorough evaluations in these domains for the…”
- Automatic Learning and Evaluation of User-Centered Objective Functions for Dialogue System Optimisation (2008, background): “…together two strands of research: one strand uses Reinforcement Learning to automatically optimise dialogue strategies, e.g. (Singh et al., 2002), (Henderson et al., 2008), (Rieser and Lemon, 2008a; Rieser and Lemon, 2008b); the other other focuses on automatic evaluation of dialogue strategies,…”
- Conversation Graph: Data Augmentation, Training, and Evaluation for Non-Deterministic Dialogue Management (2020, background): “Reinforcement learning approaches (Henderson et al., 2008; Bordes et al., 2017; Miller et al., 2017; Li et al., 2017; GordonHall et al., 2020) can replace the need for explicit training data by exploiting a custom-designed environment to infer the training signal for the policy.”
- Survey on reinforcement learning for language processing (2021, background): “Although there have been attempts to use function approximation techniques within dialogue management systems [53, 48], these have not been scaled up.”
- Feudal Reinforcement Learning for Dialogue Management in Large Domains (2018, background): “Reinforcement Learning (RL) (Sutton and Barto, 1999) has been studied for several years as a promising approach to model dialogue management (Levin et al., 1998; Henderson et al., 2008; Pietquin et al., 2011; Young et al., 2013; Casanueva et al., 2015; Su et al., 2016).”
- Datasets for Data-Driven Reinforcement Learning (2020, methodology): “Offline reinforcement learning using large datasets has also been used in real-world systems where evaluation using a simulator is not possible, such as in robotics [Cabi et al., 2019] and dialogue systems [Henderson et al., 2008, Pietquin et al., 2011, Jaques et al., 2019].”