Recursive Visual Attention in Visual Dialog

By Yulei Niu, Hanwang Zhang, Manli Zhang, Jianhong Zhang, Zhiwu Lu, Ji-rong Wen. 2018

128 citations recorded by the source corpus.

Visual dialog is a challenging vision-language task, which requires the agent to answer multi-round questions about an image. It typically needs to address two major problems: (1) How to answer visually-grounded questions, which is the core challenge in visual question answering (VQA); (2) How to infer the co-reference between questions and the dialog history. An example of visual co-reference is: pronouns (e.g., “they”) in the question (e.g., “Are they on or off?”) are linked with nouns (e.g., “lamps”) appearing in the dialog history (e.g., “How many lamps are there?”) and the object grounded in the image. In this work, to resolve the visual co-reference for visual dialog, we propose a novel attention mechanism called Recursive Visual Attention (RvA). Specifically, our dialog agent browses the dialog history until the agent has sufficient confidence in the visual co-reference resolution, and refines the visual attention recursively. The quantitative and qualitative experimental results on the large-scale VisDial v0.9 and v1.0 datasets demonstrate that the proposed RvA not only outperforms the state-of-the-art methods, but also achieves reasonable recursion and interpretable attention maps without additional annotations. The code is available at https://github.com/yuleiniu/rva.

Subfields: Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Luc Van Gool
3. Zhe Gan
4. Cordelia Schmid
5. Jianfeng Gao
6. Trevor Darrell
7. Mohit Bansal
8. Hanwang Zhang

In Question Answering & Reading Comprehension

1. Chenyou Fan
2. Devi Parikh
3. Hanwang Zhang
4. Dhruv Batra
5. Mohit Bansal
6. Yueting Zhuang
7. Jun Xiao
8. Zhe Gan

Citations in the corpus

• VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020, background): “Discriminative Setting Generative Setting MRR ↑ R@1 ↑ R@5 ↑ R@10 ↑ Mean ↓ MRR ↑ R@1 ↑ R@5 ↑ R@10 ↑ Mean ↓ LF (Das et HACAN (Yang et al., 2019), Synergistic (Guo et al., 2019a), DAN (Kang et al., 2019), and ReDAN (Gan et al., 2019).”
• Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline (2019, background): “Follow-up works on visual dialog have explored the use of deep reinforcement learning [3, 4, 17], knowledge transfer from discriminative to generative decoders [5], conditional variational autoencoders [6], generative adversarial networks [7], attention mechanisms for visual coreference resolution [”
• Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog (2019, background): “, 2019), recursive visual attention (Niu et al., 2018), factor graph attention (Schwartz et al.”
• Learning to Assemble Neural Module Tree Networks for Visual Grounding (2018, methodology): “Our module layout is trained from scratch by using the GumbelSoftmax training strategy [14], which has shown empirically effective in recent works [4, 35, 29].”
• A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, unknown): “Anaphora and co-reference resolutions are used to infer cross-references in questions and conversations [172], [173].”
• DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue (2019, background): “2017), RvA(Niu et al. 2019) and DL-61(Guo, Xu, and Tao 2019).”
• DialGraph: Sparse Graph Learning Networks for Visual Dialog (2020, methodology): “…methods that compute attended representations of the image, and the history [8,28,42,13,9,38,30], (2) a visual coreference resolution [39,25,32,19] that clarifies ambiguous expressions ( e.g ., it, them) in the question and links them to a specific entity in the image, and (3) a structural…”
• History for Visual Dialog: Do we really need it? (2020, result): “RvA: We reproduce the results of Niu et al. (2019)’s Recursive Visual Attention model (RvA), which won the 2019 VisDial challenge.”
• Dual Attention Networks for Visual Reference Resolution in Visual Dialog (2019, methodology): “Instead of relying on the visual attention maps of the previous dialogs as in (Seo et al., 2017; Kottur et al., 2018; Niu et al., 2018), we expect the FIND module to attend to the most relevant regions of the image with respect to the reference-aware representations (i.”
• Counterfactual Samples Synthesizing for Robust Visual Question Answering (2020, background): “Model fvqa typically contains an attention mechanism [13, 29, 41], and it is trained with cross-entropy loss [38, 11].”
• MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation (2022, background): “Several modeling works (Qi et al., 2020; Niu et al., 2019; Gan et al., 2019) investigate how to escalate the performance of conversational agents in image-grounded dialogue.”
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “, 2018) VGG LSTM CoAtt-GAN ADAM 3 (Niu et al., 2019) ResNet-152 LSTM RvA ADAM 7 (Zheng et al.”
• Learning to Compose Dynamic Tree Structures for Visual Contexts (2018, unknown): “In fact, the visual context serves as a powerful inductive bias that connects objects in a particular layout for high-level reasoning [26, 30, 46, 54, 36, 28].”
• Multi-View Attention Networks for Visual Dialog (2020, methodology): “(Das et al.,2017), HRE (Das et al.,2017), MN (Das et al.,2017), HCIAE (Lu et al.,2017), AMEM (Seo et al.,2017), CoAtt (Wu et al.,2018), FGA (Schwartz et al.,2019), CorefNMN (Kottur et al.,2018), RVA (Niu et al., 2019), DualVD (Jiang et al.,2020), CAG (Guo et al.,2020), HACAN (Yang et al.,2019), Syne”
• Modality-Balanced Models for Visual Dialogue (2020, background): “Visual Dialog challenge 2018 RvA (Niu et al. 2018) 55.”
• WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training (2021, background): “Second, the pre-trained text and image encoders can be directly applied to many downstream multimodal tasks such as image-to-text retrieval, text-to-image retrieval, text-to-image generation [31] and visual dialog [23].”
• Learning to Collocate Neural Modules for Image Captioning (2019, background): “Indeed, image captioning is not the only model that can easily exploit the dataset bias to captioning even without looking at the image [44], almost all existing models for visual reasoning tasks such as visual Q&A [23, 46, 48] have been spotted mode collapse to certain dataset idiosyncrasies and fa”