Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access

By Bhuwan Dhingra, Lihong Li, Xiujun Li, Jianfeng Gao, Yun-Nung (Vivian) Chen, Faisal Ahmed, Li Deng. 2016

310 citations recorded by the source corpus.

This paper proposes KB-InfoBot - a multi-turn dialogue agent which helps users search Knowledge Bases (KBs) without composing complicated queries. Such goal-oriented dialogue agents typically need to interact with an external database to access real-world knowledge. Previous systems achieved this by issuing a symbolic query to the KB to retrieve entries based on their attributes. However, such symbolic operations break the differentiability of the system and prevent end-to-end training of neural dialogue agents. In this paper, we address this limitation by replacing symbolic queries with an induced “soft” posterior distribution over the KB that indicates which entities the user is interested in. Integrating the soft retrieval process with a reinforcement learner leads to higher task success rate and reward in both simulations and against real users. We also present a fully neural end-to-end agent, trained entirely from user feedback, and discuss its application towards personalized dialogue agents.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient (2017, background): “Recently, many RL-based algorithms have been proposed and achieved good results [15, 5, 30, 22].”
Emergence of Grounded Compositional Language in Multi-Agent Populations (2017, background): “…in practical natural language applications such as machine translation (Sutskever, Vinyals, and Le 2014; Bahdanau, Cho, and Ben-gio 2014), sentiment analysis (Socher et al. 2013), document summarization (Durrett, Berg-Kirkpatrick, and Klein 2016), and domain-speciﬁc dialogue (Dhingra et al. 2016).”
SkipNet: Learning Dynamic Routing in Convolutional Networks (2017, neutral): “dress hard attention in recurrent models [4, 22].”
Key-Value Retrieval Networks for Task-Oriented Dialogue (2017, background): “While recent neural dialogue models have explicitly modelled dialogue state through belief and user intent trackers (Wen et al., 2016b; Dhingra et al., 2016; Henderson et al., 2014b), we choose instead to rely on learned neural representations for implicit modelling of dialogue state, forming 1 The…”
Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems (2018, methodology): “Dhingra et al. (2017) proposed an end-to-end RL dialogue agent for information access.”
A Survey on Dialogue Systems: Recent Advances and New Frontiers (2017, background): “[12] replaced symbolic queries with an induced “soft” posterior distribution over the knowledge base that indicates which entities the user is interested in.”
Domain Adaptive Dialog Generation via Meta Learning (2019, background): “However, our method does not get large improvement in the “Unseen slot” and “Unseen NLG” domains.”
An Overview of Natural Language State Representation for Reinforcement Learning (2020, methodology): “In Wen et al. (2017b), a sequence of free form text was mapped into a ﬁxed set of slot-value pairs by an RNN. Dhingra et al. (2017) compared a handcrafted and a neural belief tracker that uses a GRU over turns.”
Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systems (2020, methodology): “e most important factor is question type: i.e.,sent forLastFM(binaryquestion)andslen forYelp(enumerated question). The entropy(sent) information is crucial for LastFM, it is in line with the claim in [12] that the maximum entropy is the best strategy when language understanding is precise. If we abl”
Sample Efficient Deep Reinforcement Learning for Dialogue Systems With Large Action Spaces (2018, methodology): “More recently RL learning has also been applied yielding improvements [8], [9], [10], [11].”
Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog (2019, background): “…samples to train, most existing studies use data-driven approaches to build a dialog system from conversational corpora (Zhao and Eskenazi, 2016; Dhingra et al., 2017; Shah et al., 2018; Shi and Yu, 2018), where a common strategy is to build a user simulator, and then to learn dialog policy…”
Asking Clarification Questions in Knowledge-Based Question Answering (2019, background): “Dhingra et al. (2017) teach models to ask questions to limit the number of answer candidates in task-oriented dialogues.”
Deep reinforcement learning: a survey (2020, unknown): “A modified version of the episodic REINFORCE algorithm (Dhingra et al., 2017) explores and learns the policy and the posterior probability over the knowledge base entries for correct retrievals to select dialogue acts.”
Conversational Recommender System (2018, methodology): “Deep RL has been applied for better sequential decision making in various domains, including End-to-End dialogue systems using deep RL for information access [5] [32], information extraction [16], query reformulation [17], real time ads bidding [2].”
End-to-End Task-Completion Neural Dialogue Systems (2017, background): “Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 733–743, Taipei, Taiwan, November 27 – December 1, 2017 c©2017 AFNLP”