Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning

By Abhishek Das, Satwik Kottur, José M. F. Moura, Stefan Lee, Dhruv Batra. 2017

433 citations recorded by the source corpus.

We introduce the first goal-driven training for visual question answering and dialog agents. Specifically, we pose a cooperative ‘image guessing’ game between two agents – Q-BOT and A-BOT– who communicate in natural language dialog so that Q-BOT can select an unseen image from a lineup of images. We use deep reinforcement learning (RL) to learn the policies of these agents end-to-end – from pixels to multi-agent multi-round dialog to game reward.,,We demonstrate two experimental results.,,First, as a ‘sanity check’ demonstration of pure RL (from scratch), we show results on a synthetic world, where the agents communicate in ungrounded vocabularies, i.e., symbols with no pre-specified meanings (X, Y, Z). We find that two bots invent their own communication protocol and start using certain symbols to ask/answer about certain visual attributes (shape/color/style). Thus, we demonstrate the emergence of grounded language and communication among ‘visual’ dialog agents with no human supervision.,,Second, we conduct large-scale real-image experiments on the VisDial dataset [5], where we pretrain on dialog data with supervised learning (SL) and show that the RL finetuned agents significantly outperform supervised pretraining. Interestingly, the RL Q-BOT learns to ask questions that A-BOT is good at, ultimately resulting in more informative dialog and a better team.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Multimodal Learning

Citations in the corpus

Unsupervised Question Decomposition for Question Answering (2020, background): “Weakly supervised question generation often uses reinforcement learning (Nogueira & Cho, 2017; Wang & Lake, 2019; Strub et al., 2017; Das et al., 2017; Liang et al., 2018), where an unsupervised initialization can greatly mitigate the issues of exploring from scratch (Jaderberg et al., 2017).”
Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline (2019, background): “Follow-up works on visual dialog have explored the use of deep reinforcement learning [3, 4, 17], knowledge transfer from discriminative to generative decoders [5], conditional variational autoencoders [6], generative adversarial networks [7], attention mechanisms for visual coreference resolution [”
ParlAI: A Dialog Research Software Platform (2017, background): “ParlAI also supports other media, e.g. images as well as text for visual question answering (Antol et al., 2015) or visually grounded dialog (Das et al., 2017).”
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization (2016, background): “on tasks, from image classiﬁcation [33,24], object detection [21], semantic segmentation [37] to image captioning [55,7,18,29], visual question answering [3,20,42,46] and more recently, visual dialog [11, 13,12] and embodied question answering [10,23]. While 1 http://gradcam.cloudcv.org arXiv:1610.0”
Counterfactual Multi-Agent Policy Gradients (2017, background): “Also related is work on the emergence of communication between agents, learned by gradient descent (Das et al., 2017; Mordatch & Abbeel, 2017; Lazaridou et al., 2016; Foerster et al., 2016; Sukhbaatar et al., 2016).”
Learning Multi-Arm Manipulation Through Collaborative Teleoperation (2020, background): “inforcement Learning [5, 38] in cooperative settings has been widely studied [11, 12, 15, 23, 28, 37], and applied to domains such as video games [31] and visual question answering [9].”
Learning with Opponent-Learning Awareness (2017, background): “…multi-agent RL focuses on fully co-operative settings (Omidshaﬁei et al., 2017; Foerster et al., 2017a,b) and emergent communication in particular (Das et al., 2017; Mordatch and Abbeel, 2017; Lazari-dou, Peysakhovich, and Baroni, 2016; Foerster et al., 2016; Sukhbaatar, Fergus, and others,…”
Deal or No Deal? End-to-End Learning of Negotiation Dialogues (2017, background): “Similar two-stage learning strategies have been used previously (e.g. Li et al. (2016); Das et al. (2017)).”
Deep Communicating Agents for Abstractive Summarization (2018, background): “Simultaneous work has investigated the use of deep communicating agents (Sukhbaatar et al., 2016) for collaborative tasks such as logic puzzles (Foerster et al., 2016), visual dialog (Das et al., 2017), and reference games (Lazaridou et al., 2016).”
Building Generalizable Agents with a Realistic and Rich 3D Environment (2018, methodology): “On the contrary, we focus on building a platform that intersects with multiple research directions, such as object and scene understanding, 3D navigation, embodied question answering (Das et al., 2017a), while allowing users to customize the level of complexity to their needs.”
ROMA: Multi-Agent Reinforcement Learning with Emergent Roles (2020, background): “Works on the emergence of communication (Foerster et al., 2016; Lazaridou et al., 2017; Das et al., 2017; Mordatch & Abbeel, 2018; Wang et al., 2020b; Kang et al., 2020), the emergence of fairness (Jiang & Lu, 2019), and the emergence of tool usage (Baker et al.”
Look Before you Speak: Visually Contextualized Utterances (2020, background): “Popular vision and language tasks include visual question answering [4, 5, 22, 28, 58, 61], visual dialog [15, 16, 36, 40, 67], visual captioning [56, 59, 68, 73, 86], visual grounding [18, 22, 57, 83] and video-text retrieval [7, 25, 49, 53, 63].”
Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing (2018, background): “There has been a recent surge of interest in applying policy gradient methods to various application domains including program synthesis [26, 17, 68, 10], dialogue generation [25, 11], architecture search [69, 71], game [53, 31] and continuous control [44, 50].”
Reasoning Visual Dialogs With Structural and Partial Observations (2019, background): “[10] then extended [9] with an ‘image guessing’ game, i.”