Dual Attention Networks for Visual Reference Resolution in Visual Dialog

By Gi-Cheon Kang, Jaeseo Lim, Byoung-Tak Zhang. 2019

80 citations recorded by the source corpus.

Visual dialog (VisDial) is a task which requires a dialog agent to answer a series of questions grounded in an image. Unlike in visual question answering (VQA), the series of questions should be able to capture a temporal context from a dialog history and utilizes visually-grounded information. Visual reference resolution is a problem that addresses these challenges, requiring the agent to resolve ambiguous references in a given question and to find the references in a given image. In this paper, we propose Dual Attention Networks (DAN) for visual reference resolution in VisDial. DAN consists of two kinds of attention modules, REFER and FIND. Specifically, REFER module learns latent relationships between a given question and a dialog history by employing a multi-head attention mechanism. FIND module takes image features and reference-aware representations (i.e., the output of REFER module) as input, and performs visual grounding via bottom-up attention mechanism. We qualitatively and quantitatively evaluate our model on VisDial v1.0 and v0.9 datasets, showing that DAN outperforms the previous state-of-the-art model by a significant margin.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Dialogue & Conversational AI

Citations in the corpus

VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020, background): “0 dataset in the discriminative setting.”
Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog (2019, background): “, 2019), dual attention network (Kang et al., 2019), graph neural network (Zheng et al.”
DialGraph: Sparse Graph Learning Networks for Visual Dialog (2020, methodology): “…methods that compute attended representations of the image, and the history [8,28,42,13,9,38,30], (2) a visual coreference resolution [39,25,32,19] that clariﬁes ambiguous expressions ( e.g ., it, them) in the question and links them to a speciﬁc entity in the image, and (3) a structural…”
Vision-Dialog Navigation by Exploring Cross-Modal Memory (2020, methodology): “Our work is related to some visual dialogue methods [8, 19, 25, 29, 12, 37, 16] for visual coreference resolution.”
Two Causal Principles for Improving Visual Dialog (2019, background): “1) Holistic: they treat history as a whole to feed into models like HACAN [43], DAN [16] and CorefNMN [18].”
Modality-Balanced Models for Visual Dialogue (2020, result): “…et al. 2019) and MReaL– BDAI show higher NDCG scores, our consensus dropout fusion model shows more balanced results over metrics while still having a competitive NDCG score compared to DAN (Kang, Lim, and Zhang 2019), with rank 3 based on NDCG metric and high balance rank based on metric average.”
Making History Matter: History-Advantage Sequence Training for Visual Dialog (2019, methodology): “(3) Approaches that deal with visual reference resolution in VisDial (AMEM [30], CorefNMN [19], RvA [16], DAN [16]) Our method outperforms the state-of-the-art methods across most of the metrics.”
Efficient Attention Mechanism for Handling All the Interactions between Many Inputs with Application to Visual Dialog (2019, methodology): “Existing studies have considered attention from one utility to another based on different hypotheses, such as “question→ history→ image” path in [19, 27], and “question→ image → history → question” path in [13, 40], etc.”
Multi-View Attention Networks for Visual Dialog (2020, methodology): “…(Lu et al., 2017), AMEM (Seo et al., 2017), CoAtt (Wu et al., 2018), FGA (Schwartz et al., 2019), CorefNMN (Kottur et al., 2018), RVA (Niu et al., 2019), DualVD (Jiang et al., 2020), CAG (Guo et al., 2020), HACAN (Yang et al., 2019), Synergistic (Guo et al., 2019), and DAN (Kang et al., 2019).”
History for Visual Dialog: Do we really need it? (2020, background): “In contrast to previous work on the VisDial dataset, e.g. (Kottur et al., 2018; Agar-wal and Goyal, 2018; Gan et al., 2019; Guo et al., 2019; Kang et al., 2019), we are the ﬁrst to conduct a principled study of dialog history encodings.”