Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering

By Zhou Yu, Jun Yu, Chenchao Xiang, Jianping Fan, Dacheng Tao. 2017

524 citations recorded by the source corpus.

Visual question answering (VQA) is challenging, because it requires a simultaneous understanding of both visual content of images and textual content of questions. To support the VQA task, we need to find good solutions for the following three issues: 1) fine-grained feature representations for both the image and the question; 2) multimodal feature fusion that is able to capture the complex interactions between multimodal features; and 3) automatic answer prediction that is able to consider the complex correlations between multiple diverse answers for the same question. For fine-grained image and question representations, a “coattention” mechanism is developed using a deep neural network (DNN) architecture to jointly learn the attentions for both the image and the question, which can allow us to reduce the irrelevant features effectively and obtain more discriminative features for image and question representations. For multimodal feature fusion, a generalized multimodal factorized high-order pooling approach (MFH) is developed to achieve more effective fusion of multimodal features by exploiting their correlations sufficiently, which can further result in superior VQA performance as compared with the state-of-the-art approaches. For answer prediction, the Kullback–Leibler divergence is used as the loss function to achieve precise characterization of the complex correlations between multiple diverse answers with the same or similar meaning, which can allow us to achieve faster convergence rate and obtain slightly better accuracy on answer prediction. A DNN architecture is designed to integrate all these aforementioned modules into a unified model for achieving superior VQA performance. With an ensemble of our MFH models, we achieve the state-of-the-art performance on the large-scale VQA data sets and win the runner-up in VQA Challenge 2017.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019, neutral): “VQA The SotA result is BAN+Counter in Kim et al. (2018), which achieves the best accuracy among other recent works: MFH (Yu et al., 2018), Pythia (Jiang et al., 2018), DFAF (Gao et al., 2019a), and Cycle-Consistency (Shah et al., 2019).5 LXMERT improves the SotA overall accuracy (‘Accu’ in Table 2) ”
Towards VQA Models That Can Read (2019, methodology): “The fQ module passes GloVe embeddings to an LSTM [15] with self-attention [49] to generate question’s sentence embedding.”
Bilinear Attention Networks (2018, methodology): “Yu et al. [39] extend low-rank bilinear pooling [15] with the rank k > 1 and two factorized three-dimensional matrices, which called as MFB.”
Deep Modular Co-Attention Networks for Visual Question Answering (2019, methodology): “We compare MCAN ed -6 with the best two approaches (MFH [33] and BAN-8 [14]) in Table 4 in terms of overall accuracy ± std, number of parameters and FLOPs, respectively.”
Pythia v0.1: the Winning Entry to the VQA Challenge 2018 (2018, methodology): “To compute the question representation, we used 300D GloVe [11] vectors to initialize the word embeddings and then passed it to a GRU network and a question attention module to extract attentive text features [16].”
In Defense of Grid Features for Visual Question Answering (2020, methodology): “We use the co-attention model [51] implemented in Pythia [16, 36].”
Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering (2018, background): “Multi-modal Factorized High-order Pooling (MFH) [43] is a state-of-the-art bilinear pooling methods.”
Learning to Count Objects in Natural Images for Visual Question Answering (2018, methodology): “Despite our single-model baseline being substantially worse than the state-of-the-art, by simply adding the counting component we outperform even the 8-model ensemble in Zhou et al. (2017) on the number category.”
Multimodal Unified Attention Networks for Vision-and-Language Interactions (2019, methodology): “reduced the co-attention method into two steps, self-attention for a question embedding and the question-conditioned attention for a visual embedding [35].”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, background): “to learn joint representations [174], [175].”
ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering (2019, background): “…many approaches proposed to investigate the key components of this task, e.g. , discriminative feature representation (Anderson et al. 2018), multi-modal fusion (Kim et al. 2017a; Yu et al. 2017; Yu et al. 2018a) and visual reasoning (Nam, Ha, and Kim 2016; Lu et al. 2016; Johnson et al. 2017b).”
Relation-Aware Graph Attention Network for Visual Question Answering (2019, background): “[15, 25, 60, 4, 24] explored higherorder fusion methods to better combine textual information with visual information (e.”
Deep Multimodal Data Fusion (2024, background): “There are various other multimodal tasks, e.g., visual question answering ( VQA ) (text-image: [154, 220], text-video: [82, 223]), RGB-depth object segmentation [31, 39], medical data analysis [181, 185], and image captioning [216, 237].”
ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration (2021, background): “, visual question answering [50, 51], image text retrieval [18, 31], and referring expression comprehension [16, 52].”
Learning Answer Embeddings for Visual Question Answering (2018, background): “(7) has been exploited in several recent work [30, 12, 15] to boost the performance on VQA [3] and VQA2 [9].”
Multi-Modality Latent Interaction Network for Visual Question Answering (2019, methodology): “Previous approaches solve the Visual Question Answering (VQA) by designing better features [25, 44, 13, 17, 1], better bilinear fusion approaches [10, 7, 22, 3, 52] or better attention mechanisms [48, 29, 49, 45, 36].”