End-to-end LSTM-based dialog control optimized with supervised and reinforcement learning

By Jason Williams, Geoffry Zweig. 2016

159 citations recorded by the source corpus.

This paper presents a model for end-to-end learning of task-oriented dialog systems. The main component of the model is a recurrent neural network (an LSTM), which maps from raw dialog history directly to a distribution over system actions. The LSTM automatically infers a representation of dialog history, which relieves the system developer of much of the manual feature engineering of dialog state. In addition, the developer can provide software that expresses business rules and provides access to programmatic APIs, enabling the LSTM to take actions in the real world on behalf of the user. The LSTM can be optimized using supervised learning (SL), where a domain expert provides example dialogs which the LSTM should imitate; or using reinforcement learning (RL), where the system improves by interacting directly with end users. Experiments show that SL and RL are complementary: SL alone can derive a reasonable initial policy from a small number of training dialogs; and starting RL optimization with a policy trained with SL substantially accelerates the learning rate of RL.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Efficient Natural Language Response Suggestion for Smart Reply (2017, background): “In recent work, the components of task-oriented dialog systems have been implemented as neural networks, enabling joint learning of robust models [7, 26, 27].”
Neural Approaches to Conversational AI (2018, positive): “n expert policies that produce responses for supervised training. This inspire another line of work that uses reinforcement learning to optimize end-to-end dialogue systems (Zhao and Esk´enazi, 2016; Williams and Zweig, 2016; Dhingra et al., 2017; Li et al., 2017d; Braunschweiler and Papangelis, 201”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, background): “Researchers have also proposed training dialogue systems online through live interaction with humans, and offline using user simulator models and reinforcement learning methods (Levin et al., 1997; Georgila et al., 2006; Paek, 2006; Schatzmann et al., 2007; Jung et al., 2009; Schatzmann and Young, 2”
End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient (2017, methodology): “Usually, these algorithms are followed by training with reinforcement learning to further refine the model [21, 28, 23, 27].”
Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems (2018, neutral): “Comparing to SL models, systems trained with RL by receiving feedback during users interactions showed improved model robustness against diverse dialogue scenarios (Williams and Zweig, 2016; Liu and Lane, 2017b).”
BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems (2016, positive): “red by recent breakthroughs on various domains such as Atari games [Mnih et al., 2013], board games [Silver et al., 2016] and some promising results on dialog [Fatemi et al., 2016, Cuayahuitl, 2016, Williams and Zweig, 2016], we use deep reinforcement learning (DRL), an approach that weds the repres”
Frames: a corpus for adding memory to goal-oriented dialogue systems (2017, methodology): “Much work has focused on spoken dialogue (Lemon and Pietquin, 2007; Walker et al., 1998; Williams and Zweig, 2016), since spoken dialogue systems are useful in many settings, including in hands-free environments such as cars (Lemon et al., 2006).”
Iterative policy learning in end-to-end trainable task-oriented neural dialog models (2017, background): “In many of the recent work on using RL for dialog policy learning [19, 31, 20], hand-designed user simulators are used to interact with the dialog agent.”
A Survey on Dialogue Systems: Recent Advances and New Frontiers (2017, methodology): “Previous systems achieved this by issuing a symbolic query to the knowledge base to retrieve entries based on their attributes, where semantic parsing on the input is performed to construct a symbolic query representing the beliefs of the agent about the user goal[84; 90; 36].”
Sample Efficient Deep Reinforcement Learning for Dialogue Systems With Large Action Spaces (2018, background): “A number of recent works investigated using deep reinforcement learning in an end-to-end stetting [12], [27], where belief tracking and policy optimisation are optimised jointly.”
An End-to-end Approach for Handling Unknown Slot Values in Dialogue State Tracking (2018, background): “It is worth noting that end-to-end (E2E) modeling for task-oriented dialogue systems has become a popular trend (Williams and Zweig, 2016; Zhao and Eskenazi, 2016; Li et al., 2017; Liu et al., 2017; Wen et al., 2017), although most of them focus on E2E policy learning and language generation, and…”
Modeling Multi-Action Policy for Task-Oriented Dialogues (2019, background): “Both supervised learning (SL) (Stent, 2002; Williams et al., 2017a; Williams and Zweig, 2016; Henderson et al., 2005, 2008) and reinforcement learning (RL) approaches (Walker, 2000; Young et al.”
Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access (2016, methodology): “Neural Policy Network: For the neural approach, similar to (Williams and Zweig, 2016; Zhao and Eskenazi, 2016), we use an RNN to allow the network to maintain an internal state of dialogue history.”
Natural Language Processing Advancements By Deep Learning: A Survey (2020, methodology): “Some approaches to this challenge include converting the user input into internal representations [166], combining supervised and reinforced learning [167], and extending the memory network approach [168] for question-answering to a dialog”
Fine Grained Knowledge Transfer for Personalized Task-oriented Dialogue Systems (2017, background): “Williams et al. (Williams and Zweig 2016; Bordes and Weston 2016) used learning meth-ods in spoken language understanding and dialogue management.”
Advancing Biosensors with Machine Learning. (2020, background): “102,103 − 106 Recent studies have achieved promising brain tumor segmentation performance by training RNN. 107 Long Short-Term Memory (LSTM) networks are a special type of RNN with a capability for long-term dependencies.”
Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models (2019, background): “Training with RL usually has 2 steps: supervised pre-training and policy gradient reinforcement learning (Williams and Zweig, 2016; Dhingra et al., 2017; Li et al., 2016).”
End-to-End Reinforcement Learning of Dialogue Agents for Information Access (2016, methodology): “An alternative way for the dialogue agent to interface with the database is by augmenting its action space with predeﬁned API calls (Williams and Zweig, 2016; Zhao and Eskenazi, 2016; Bordes and Weston, 2016).”