Visual Reference Resolution using Attention Memory for Visual Dialog

By Paul Hongsuck Seo, Andreas M. Lehrmann, Bohyung Han, Leonid Sigal. 2017

125 citations recorded by the source corpus.

Visual dialog is a task of answering a series of inter-dependent questions given an input image, and often requires to resolve visual references among the questions. This problem is different from visual question answering (VQA), which relies on spatial attention (a.k.a. visual grounding) estimated from an image and question pair. We propose a novel attention mechanism that exploits visual attentions in the past to resolve the current reference in the visual dialog scenario. The proposed model is equipped with an associative attention memory storing a sequence of previous (attention, key) pairs. From this memory, the model retrieves the previous attention, taking into account recency, which is most relevant for the current question, in order to resolve potentially ambiguous references. The model then merges the retrieved attention with a tentative one to obtain the final attention for the current question; specifically, we use dynamic parameter prediction to combine the two attentions conditioned on the question. Through extensive experiments on a new synthetic visual dialog dataset, we show that our model significantly outperforms the state-of-the-art (by ~16 % points) in situations, where visual reference resolution plays an important role. Moreover, the proposed model achieves superior performance (~ 2 % points improvement) in the Visual Dialog dataset, despite having significantly fewer parameters than the baselines.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020, background): “In general, there are two types of decoder to predict the answer: a discriminative decoder that ranks the answer candidates and is trained with a cross entropy loss, or a generative decoder that synthesizes an answer and is trained with a maximum log-likelihood loss.”
Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog (2019, methodology): “[39] solves visual coreference resolution implicitly via attention memory, while [26] solves the same problem, but using a more explicit reasoning procedure based on neural module networks [2].”
Look Before you Speak: Visually Contextualized Utterances (2020, background): “Popular vision and language tasks include visual question answering [4, 5, 22, 28, 58, 61], visual dialog [15, 16, 36, 40, 67], visual captioning [56, 59, 68, 73, 86], visual grounding [18, 22, 57, 83] and video-text retrieval [7, 25, 49, 53, 63].”
Two Causal Principles for Improving Visual Dialog (2019, methodology): “All of the existing approaches in the VisDial task are based on the typical encoder-decoder framework [15, 12, 33, 11, 32, 47].”
Large-Scale Answerer in Questioner's Mind for Visual Dialog Question Generation (2019, methodology): “Retrieval-based models, which are basically discriminative models that select a response from a predeﬁned candidate set of system responses, are often used in task-oriented dialog tasks (Bordes & Weston, 2017; Seo et al., 2017a; Liu & Perez, 2017).”
Factor Graph Attention (2019, background): “In contrast, classical methods [52, 44] attend over history locations only.”
DialGraph: Sparse Graph Learning Networks for Visual Dialog (2020, methodology): “…methods that compute attended representations of the image, and the history [8,28,42,13,9,38,30], (2) a visual coreference resolution [39,25,32,19] that clariﬁes ambiguous expressions ( e.g ., it, them) in the question and links them to a speciﬁc entity in the image, and (3) a…”
Semantic bottleneck for computer vision tasks (2018, background): “Another way to enrich the caption is to generate a set of questions/answers such as proposed in the Visual Dialog framework [1,13,14].”
Making History Matter: History-Advantage Sequence Training for Visual Dialog (2019, methodology): “(3) Approaches that deal with visual reference resolution (AMEM [32], CorefNMN [20], RvA [17], DAN [17]), the methods focus on explicit visual co-reference resolution in visual dialog.”
Making History Matter: Gold-Critic Sequence Training for Visual Dialog (2019, background): “A good dialog agent should accomplish a complex series of reasoning sub-tasks: contextual visual perception [31, 21, 6, 28, 18, 32, 29], language modeling [15, 5], and co-reference resolution in dia-Figure log history [28, 18] ( e.g ., identify what is “that”).”
Recursive Visual Attention in Visual Dialog (2018, methodology): “Different from [27, 17] that ignored the continuity of visual attention in the dialog, we formulate visual co-reference resolution in visual dialog as Recursive Visual Attention (RvA).”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “, 2017) VGG LSTM HCIAE-NP-ATT ADAM 7 (Seo et al., 2017) VGG LSTM AMEM ADAM 7 (Jain et al.”
Multi-View Attention Networks for Visual Dialog (2020, methodology): “9 datasets for the following methods: LF (Das et al., 2017), HRE (Das et al., 2017), MN (Das et al., 2017), HCIAE (Lu et al., 2017), AMEM (Seo et al., 2017), CoAtt (Wu et al., 2018), FGA (Schwartz et al., 2019), CorefNMN (Kottur et al., 2018), RVA (Niu et al., 2019), DualVD (Jiang et al., 2020), CAG”
CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog (2019, methodology): “(3) MNIST-Dialog (Seo et al., 2017) is a synthetic dialog dataset on a grid of 4×4 stylized MNIST digits (Fig.”
Visual Coreference Resolution in Visual Dialog using Neural Module Networks (2018, methodology): “We show quantitative benefits of these modules on a reasoning-wise complicated but visually simple MNIST dialog dataset [41], where achieve near perfect accuracy.”