Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems

By Bing Liu, Gökhan Tür, Dilek Z. Hakkani-Tür, Pararth Shah, Larry Heck. 2018

176 citations recorded by the source corpus.

In this work, we present a hybrid learning method for training task-oriented dialogue systems through online user interactions. Popular methods for learning task-oriented dialogues include applying reinforcement learning with user feedback on supervised pre-training models. Efficiency of such learning method may suffer from the mismatch of dialogue state distribution between offline training and online interactive learning stages. To address this challenge, we propose a hybrid imitation and reinforcement learning method, with which a dialogue agent can effectively learn from its interaction with users by learning from human teaching and feedback. We design a neural network based task-oriented dialogue agent that can be optimized end-to-end with the proposed learning method. Experimental results show that our end-to-end dialogue agent can learn effectively from the mistake it makes via imitation learning from user teaching. Applying reinforcement learning with user feedback after the imitation learning stage further improves the agent’s capability in successfully completing a task.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Graph-based, Self-Supervised Program Repair from Diagnostic Feedback (2020, background): “…repair in this paper, we note that our framework of learning to edit based on feed-back is a potentially powerful and more general paradigm with many applications, from learning to edit essays based on written feedback, to learning from users in interactive dialogue, etc. (Liu et al., 2018).”
Learning from Dialogue after Deployment: Feed Yourself, Chatbot! (2019, methodology): “requested, existing methods typically require accompanying scalar rewards or adherence to particular templates or structure to ensure that the feedback is usable by the model (Rieser and Lemon, 2011; Zhang et al., 2017; Liu et al., 2018).”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “Improving dialog systems with RL has largely been restricted to task-oriented dialog systems, which have a limited number of task-specific actions (Fatemi et al., 2016; Gašić et al., 2011; Liu and Lane, 2017; Liu et al., 2018; Su et al., 2017).”
A Simple Language Model for Task-Oriented Dialogue (2020, methodology): “[28] proposed a hybrid imitation and reinforcement learning method, by jointly learning a policy for dialogue management with response generation.”
Neural Approaches to Conversational AI (2018, neutral): “This inspire another line of work that uses reinforcement learning to optimize end-to-end dialogue systems (Zhao and Eskénazi, 2016; Williams and Zweig, 2016; Dhingra et al., 2017; Li et al., 2017d; Braunschweiler and Papangelis, 2018; Strub et al., 2017; Liu and Lane, 2017; Liu et al., 2018a).”
MOSS: End-to-End Dialog System Framework with Modular Supervision (2019, background): “There is also work that incorporates supervision from dialog policy but not natural language understanding (Liu et al. 2018).”
End-to-End Learning of Task-Oriented Dialogs (2018, methodology): “With experiments (Liu et al., 2018) in a movie booking domain, we show that the agent can efficiently learn from the expert demonstrations and improve dialog task success rate with the proposed imitation”
Improving Factual Consistency Between a Response and Persona Facts (2021, methodology): “RL has been extensively used for training task-oriented dialogue systems (e.g., Nogueira and Cho (2017); Liu et al. (2018)).”
Incorporating the Structure of the Belief State in End-to-End Task-Oriented Dialogue Systems (2018, background): “Our work is also akin to modularly connected end-to-end trainable networks [23, 22, 13, 12, 11].”
Conversation Graph: Data Augmentation, Training, and Evaluation for Non-Deterministic Dialogue Management (2020, background): “ConvGraph can be efficiently expanded as the environment or user behaviours change over time in order to extend an artificial agent with additional capabilities or to bootstrap agent policies interactively (Williams and Liden, 2017; Williams et al., 2017; Liu et al., 2018).”
Dynamic Planning in Open-Ended Dialogue using Reinforcement Learning (2022, methodology): “[23] encode the dialogue state using an RNN but translate the representation to slot-value pairs.”
ConveRT: Efficient and Accurate Conversational Representations from Transformers (2019, background): “Transfer learning on top of pretrained models (Devlin et al., 2019; Liu et al., 2019, inter alia) provides one avenue for reducing the amount of annotated data required to train models capable of generalization.”
Dialogue in the Wild: Learning from a Deployed Role-Playing Game with Humans and Bots (2021, methodology): “Liu et al. (2018) applied such an approach to goal-oriented tasks.”
Survey on evaluation methods for dialogue systems (2019, background): “Furthermore, the pipelined architecture leads to the propagation and amplification of errors through the pipeline as each module depends on the output of the previous module (Li et al., 2017b; Liu et al., 2018).”
Adversarial Learning of Task-Oriented Neural Dialog Models (2018, background): “Recent efforts have been made in designing end-to-end solutions (Williams and Zweig, 2016; Liu and Lane, 2017a; Li et al., 2017b; Liu et al., 2018) for task-oriented dialogs.”