Factor Graph Attention

By Idan Schwartz, Seunghak Yu, Tamir Hazan, Alexander Gerhard Schwing. 2019

115 citations recorded by the source corpus.

Dialog is an effective way to exchange information, but subtle details and nuances are extremely important. While significant progress has paved a path to address visual dialog with algorithms, details and nuances remain a challenge. Attention mechanisms have demonstrated compelling results to extract details in visual question answering and also provide a convincing framework for visual dialog due to their interpretability and effectiveness. However, the many data utilities that accompany visual dialog challenge existing attention techniques. We address this issue and develop a general attention mechanism for visual dialog which operates on any number of data utilities. To this end, we design a factor graph based attention mechanism which combines any number of utility representations. We illustrate the applicability of the proposed approach on the challenging and recently introduced VisDial datasets, outperforming recent state-of-the-art methods by 1.1% for VisDial0.9 and by 2% for VisDial1.0 on MRR. Our ensemble model improved the MRR score on VisDial1.0 by more than 6%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhe Gan
2. Jianfeng Gao
3. Mohit Bansal
4. Jingjing Liu
5. Linjie Li
6. Yu Cheng
7. Caiming Xiong
8. Devi Parikh

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Luc Van Gool
3. Zhe Gan
4. Trevor Darrell
5. Xiaogang Wang
6. Cordelia Schmid
7. Jianfeng Gao
8. Yi Yang

In Model Architectures

1. Zhe Gan
2. Hanwang Zhang
3. Yu Cheng
4. Jingjing Liu
5. Linjie Li
6. Devi Parikh
7. Steven C. H. Hoi
8. Caiming Xiong

Citations in the corpus

• VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020, methodology): “However, their model neglects the important early interaction of the answer entity and cannot naturally leverage the pretrained language representations from BERT like ours.”
• Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline (2019, background): “Recent years have seen incredible progress in Visual Dialog [1, 5–25], spurred in part by the initial efforts of Das et al.”
• Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog (2019, background): “, 2018), factor graph attention (Schwartz et al., 2019), dual attention network (Kang et al.”
• DialGraph: Sparse Graph Learning Networks for Visual Dialog (2020, methodology): “Typically, most of the previous research has focused on extracting the rich question-relevant representations from the given image and dialog history, while implicitly finding their relationships [8,28,42,13,9,38].”
• Learning to Ground Visual Objects for Visual Dialog (2021, methodology): “FGA (Schwartz et al., 2019) realizes a factor graph attention mechanism, which constructs the graph over all the multi-modal features and estimates their interactions to ground visual objects.”
• Multi-View Attention Networks for Visual Dialog (2020, background): “Another direction of research that is inspired by graphical networks focuses on learning the inherent relations among image, dialog history, and question (Zheng et al. 2019; Schwartz et al. 2019; Guo et al. 2020).”
• Efficient Attention Mechanism for Handling All the Interactions between Many Inputs with Application to Visual Dialog (2019, methodology): “Motivated by this, a recent study tries to capture all the possible interactions by using a factor graph [34].”
• Vision-Dialog Navigation by Exploring Cross-Modal Memory (2020, methodology): “Our work is related to some visual dialogue methods [8, 19, 25, 29, 12, 37, 16] for visual coreference resolution.”
• ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic (2021, background): “Attention was added to identify relevant salient objects [59, 61, 71].”
• Iterative Context-Aware Graph Inference for Visual Dialog (2020, methodology): “2020.01007 models considered the fixed graph attention or embedding, such as fixed fully-connected graph (FGA [28]), fixed once graph attention evolution (FGA [28]) and fixed unidirectional message passing (GNN [42]).”
• DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue (2019, result): “DL-61 (Guo, Xu, and Tao 2019) is a two-stage network for candidate selection and re-ranking while FGA (Schwartz et al. 2019) conducts attention across all the data parts, which gain relatively high performance on some metrics compared with our model.”
• Two Causal Principles for Improving Visual Dialog (2019, methodology): “All of the existing approaches in the VisDial task are based on the typical encoder-decoder framework [15, 12, 33, 11, 32, 48].”