End-to-end optimization of goal-driven and visually grounded dialogue systems

By Florian Strub, Harm de Vries, Jérémie Mary, Bilal Piot, Aaron C. Courville, Olivier Pietquin. 2017

141 citations recorded by the source corpus.

End-to-end design of dialogue systems has recently become a popular research topic thanks to powerful tools such as encoder-decoder architectures for sequence-to-sequence learning. Yet, most current approaches cast human-machine dialogue management as a supervised learning problem, aiming at predicting the next utterance of a participant given the full history of the dialogue. This vision is too simplistic to render the intrinsic planning problem inherent to dialogue as well as its grounded nature , making the context of a dialogue larger than the sole history. This is why only chitchat and question answering tasks have been addressed so far using end-to-end architectures. In this paper, we introduce a Deep Reinforcement Learning method to optimize visually grounded task-oriented dialogues , based on the policy gradient algorithm. This approach is tested on a dataset of 120k dialogues collected through Mechanical Turk and provides encouraging results at solving both the problem of generating natural dialogues and the task of discovering a specific object in a complex picture.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Learning from Dialogue after Deployment: Feed Yourself, Chatbot! (2019, background): “Learning to ask questions is another area that has been studied (Strub et al., 2017; Wang et al., 2018; Rao and III, 2018).”
Unsupervised Question Decomposition for Question Answering (2020, background): “Weakly supervised question generation often uses reinforcement learning (Nogueira & Cho, 2017; Wang & Lake, 2019; Strub et al., 2017; Das et al., 2017; Liang et al., 2018), where an unsupervised initialization can greatly mitigate the issues of exploring from scratch (Jaderberg et al., 2017).”
Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline (2019, background): “Follow-up works on visual dialog have explored the use of deep reinforcement learning [6, 7, 20], knowledge transfer from discriminative to generative decoders [8], conditional variational autoencoders [9], generative adversarial networks [10], attention mechanisms for visual coreference resolution ”
Neural Approaches to Conversational AI (2018, positive): “This inspire another line of work that uses reinforcement learning to optimize end-to-end dialogue systems (Zhao and Eskénazi, 2016; Williams and Zweig, 2016; Dhingra et al., 2017; Li et al., 2017d; Braunschweiler and Papangelis, 2018; Strub et al., 2017; Liu and Lane, 2017; Liu et al., 2018a).”
Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog (2019, methodology): “For the GuessWhat?! task, various methods such as RL have been proposed to improve the performance of dialog agents, measured by task completion rate as in goal-oriented dialog [46, 40, 47, 28, 57].”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “However, other approaches constrained themselves to a domain proposing end-to-end optimization (Strub et al., 2017).”
What Should I Ask? Using Conversationally Informative Rewards for Goal-oriented Visual Dialog. (2019, unknown): “Table 1: A comparison of the recognition accuracy of our model with the state of the art model (Strub et al., 2017) and other concurrent models on the GuessWhat?! task for guessing an object in the images from the test set.”
Interactive Language Acquisition with One-shot Visual Concept Learning through a Conversational Game (2018, background): “…cannot speak (Wang et al., 2016) or “speaks” by selecting a candidate from a provided set (He et al., 2016; Weston, 2016; Li et al., 2017) rather than generating sentences by itself, or games mainly focus on slow learning (Das et al., 2017; Strub et al., 2017) and falls short on one-shot learning.”
Best of Both Worlds: Transferring Knowledge from Discriminative Learning to a Generative Visual Dialog Model (2017, background): “[47, 44, 46]) have emerged as the dominant paradigm across a variety of setting and datasets – from text-only dialog [44, 40, 23, 3] to more recently, visual dialog [7, 9, 8, 33, 45], where an agent must answer a sequence of questions grounded in an image, requiring it to reason about both visual co”
Talk the Walk: Navigating New York City through Grounded Dialogue (2018, background): “…communication, where, agents communicate (in natural language or otherwise) in order to solve a task, with respect to their shared environment (Lazaridou et al., 2016; Das et al., 2017b; Mordatch & Abbeel, 2017; Evtimova et al., 2017; Lewis et al., 2017; Strub et al., 2017; Kottur et al., 2017).”
Beyond task success: A closer look at jointly learning to see, ask, and GuessWhat (2018, unknown): “Reinforcement learning for visual dialogue agents was introduced by Das et al. (2017b) for VisDial and by Strub et al. (2017) for GuessWhat?!.”
Learning Natural Language Generation with Truncated Reinforcement Learning (2022, methodology): “During this process, the agent may be rewarded with language scores (Ran-zato et al., 2016), human preferences (Stiennon et al., 2020) or task completion scores (Strub et al., 2017).”
Large-Scale Answerer in Questioner's Mind for Visual Dialog Question Generation (2019, methodology): “To address this issue, RL has been applied to the problem (Strub et al., 2017; Das et al., 2017b).”
Adversarial Language Games for Advanced Natural Language Intelligence (2019, background): “, infer the intention of the partner in contexts) of language when agents share a common goal (Lazaridou et al., 2017; Strub et al., 2017; He et al., 2017; Khani et al., 2018).”