Structured Attentions for Visual Question Answering

By Chen Zhu, Yanpeng Zhao, Shuaiyi Huang, Kewei Tu, Yi Ma. 2017

112 citations recorded by the source corpus.

Visual attention, which assigns weights to image regions according to their relevance to a question, is considered as an indispensable part by most Visual Question Answering models. Although the questions may involve complex rela- tions among multiple regions, few attention models can ef- fectively encode such cross-region relations. In this paper, we demonstrate the importance of encoding such relations by showing the limited effective receptive field of ResNet on two datasets, and propose to model the visual attention as a multivariate distribution over a grid-structured Con- ditional Random Field on image regions. We demonstrate how to convert the iterative inference algorithms, Mean Field and Loopy Belief Propagation, as recurrent layers of an end-to-end neural network. We empirically evalu- ated our model on 3 datasets, in which it surpasses the best baseline model of the newly released CLEVR dataset [13] by 9.5%, and the best published model on the VQA dataset [3] by 1.25%. Source code is available at https://github.com/zhuchen03/vqa-sva.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “There have also been numerous visual reasoning models (Hudson & Manning, 2018; Santoro et al., 2017; Hu et al., 2017; Perez et al., 2018; Zhu et al., 2017; Mascharka et al., 2018; Suarez et al., 2018; Cao et al., 2018; Bisk et al., 2018; Misra et al., 2018; Aditya et al., 2018).”
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding (2018, background): “There have been other end-to-end neural models that has achieved nice performance on the dataset, exploiting various attention structures and accounting object relations [Hudson and Manning, 2018, Santoro et al., 2017, Hu et al., 2017, Perez et al., 2018, Zhu et al., 2017].”
Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering (2018, background): “This approach has achieved great success in Natural Language Processing (NLP) [3], image captioning [40] and VQA [46].”
Learning to Count Objects in Natural Images for Visual Question Answering (2018, background): “Hard attention (Ba et al., 2015; Mnih et al., 2014) and structured attention (Kim et al., 2017) may be possible solutions to this, though no signiﬁcant improvement in counting ability has been found for the latter so far (Zhu et al., 2017).”
Learning Visual Question Answering by Bootstrapping Hard Attention (2018, neutral): “-only [34,22,23,33,41] 46.8 41.7 61.1 69.8 36.8 51.8 CNN+LSTM [34,22,23,33,41] 52.3 43.7 65.2 67.1 49.3 53.0 SAN [34,9] 68.5 52.2 71.1 73.5 85.3 52.3 SAN* [25,9] 76.6 64.4 82.7 77.4 82.6 75.4 LBP-SIG [68] 78.0 61.3 79.6 80.7 88.6 76.3 N2NMN [69] 83.7 68.5 85.7 85.0 90.0 88.9 PG+EE (700k) [70] 96.9 9”
Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning (2018, background): “[32] aptly note that common VQA architectures have receptive fields that are too small for this global information propagation.”
Motion-Appearance Co-memory Networks for Video Question Answering (2018, neutral): “Many methods have been proposed [37, 5, 14, 21, 36, 7, 3, 1, 33, 23, 4, 28, 17, 13, 40, 38, 41, 42].”
Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism (2019, methodology): “References [29] and [27] employed this framework for visual question answering and image caption respectively.”
Improved Fusion of Visual and Language Representations by Dense Symmetric Co-attention for Visual Question Answering (2018, methodology): “[4] proposed a structured attention model that can encode cross-region relation, aiming at properly answering questions that involve complex inter-region relations.”
Interpretable Visual Question Answering by Reasoning on Dependency Trees (2018, background): “in [5], [44] to extract image features and word encodings.”
Multi-Modality Latent Interaction Network for Visual Question Answering (2019, background): “Structured attention [55] added a Markov Random Field (MRF) model over the spatial attention map for modelling spatial importance.”
Meta Module Network for Compositional Visual Reasoning (2019, background): “Most monolithic networks for visual reasoning resort to attention mechanism for multimodal fusion [49, 50, 48, 45, 46, 24, 23, 22, 26, 16].”
Making History Matter: History-Advantage Sequence Training for Visual Dialog (2019, background): “A number of approaches[39, 40, 5, 11, 33] have been proposed that apply question-based attention module on image features.”
Making History Matter: Gold-Critic Sequence Training for Visual Dialog (2019, methodology): “[33] developed the attention model which can encode the cross-region relationships and support the model to answer more complex questions with visual reference resolution.”
Deep Multimodal Representation Learning: A Survey (2019, background): “[161] proposed a structured attention model to capture the semantic structure among image regions, and their experiments showed that this model is capable of inferring spatial relations and attending to the right region.”
CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval (2019, background): “Different types of interactions have been explored in language and vision tasks beyond text-image retrieval [32, 2, 20, 35, 12, 29, 21, 18, 19].”
RAVEN: A Dataset for Relational and Analogical Visual REasoNing (2019, background): “Modules with special attention mechanism were latter proposed in an end-to-end manner to solve this visual reasoning task [19, 49, 59].”
Factor Graph Attention (2019, background): “While attention mechanisms have been applied to visual question answering [14, 33, 42, 26, 55, 58], few works have addressed visual dialog because of the many different data utilities.”
MUREL: Multimodal Relational Reasoning for Visual Question Answering (2019, methodology): “Instead, state-of-the-art methods often rely on the much simpler attentional framework [11, 8, 20, 10].”
Relation-Aware Graph Attention Network for Visual Question Answering (2019, background): “In lieu of directly using visual features from CNN-based feature extractors, [56, 11, 41, 33, 49, 38, 63, 36] explored various image attention mechanisms to locate regions that are relevant to the question.”