Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering

By Zhou Yu, Jun Yu, Jianping Fan, Dacheng Tao. 2017

741 citations recorded by the source corpus.

Visual question answering (VQA) is challenging because it requires a simultaneous understanding of both the visual content of images and the textual content of questions. The approaches used to represent the images and questions in a fine-grained manner and questions and to fuse these multimodal features play key roles in performance. Bilinear pooling based models have been shown to outperform traditional linear models for VQA, but their high-dimensional representations and high computational complexity may seriously limit their applicability in practice. For multimodal feature fusion, here we develop a Multi-modal Factorized Bilinear (MFB) pooling approach to efficiently and effectively combine multi-modal features, which results in superior performance for VQA compared with other bilinear pooling approaches. For fine-grained image and question representation, we develop a ‘co-attention’ mechanism using an end-to-end deep network architecture to jointly learn both the image and question attentions. Combining the proposed MFB approach with co-attention learning in a new network architecture provides a unified model for VQA. Our experimental results demonstrate that the single MFB with co-attention model achieves new state-of-theart performance on the real-world VQA dataset. Code available at https://github.com/yuzcccc/mfb.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “Prominent work includes: (i) advanced attention mechanisms [40, 79, 51]; (ii) better multimodal fusion methods [14, 80, 27, 26]; (iii) multi-step reasoning [77, 22, 15]; (iv) incorporation of object relations [55, 48, 6, 32]; and (v) neural module networks for compositional reasoning [3, 25, 20, 11]”
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, background): “– Various bilinear pooling approaches such as compact [13], low-rank [28], factorized [58], etc.”
Deep Modular Co-Attention Networks for Visual Question Answering (2019, neutral): “[32, 33] and Ben et al.”
Hierarchical Conditional Relation Networks for Video Question Answering (2020, background): “Earlier attempts for generic multimodal fusion for visual reasoning includes bilinear operators, either applied directly [13] or through attention [13, 43].”
Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog (2019, methodology): “Now, we use Multimodal Factorized Bilinear pooling (MFB) (Yu et al., 2017c) to fuse vt and dt together.”
Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning (2018, background): “While a wide variety of models have recently been proposed for the VQA task [6, 10, 19, 21, 28, 30], neural module networks [2, 3, 10, 14] are among the most intuitive.”
Multimodal Transformer With Multi-View Visual Representation for Image Captioning (2019, background): “visual question answering [2]–[4], visual grounding [5] and”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, background): “to learn joint representations [174], [175].”
Bilinear Graph Networks for Visual Question Answering (2019, background): “The Hadamard product for low-rank bilinear pooling (MLB) [10] models the common vector with a low-rank matrix by an elementwise multiplication, and multimodal factorized bilinear pooling (MFB) [24] increases the rank from 1 to k to accelerate the convergence rate and improve the model’s robustness.”
Quantifying and Alleviating the Language Prior Problem in Visual Question Answering (2019, background): “Derived from it, other QA systems like community QA (CQA) [29], multimedia QA [28]and visual QA (VQA) [5, 13, 41] have attracted researchers’ interest in recent years.”
Interpretable Visual Question Answering by Visual Grounding From Attention Supervision Mining (2018, methodology): “We build the attention supervision on top of the opensourced implementation of MCB [6] and MFB [25].”
ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration (2021, background): “, visual question answering [50, 51], image text retrieval [18, 31], and referring expression comprehension [16, 52].”
Interpretable Visual Question Answering by Reasoning on Dependency Trees (2018, background): “Among these works, the attention mechanism [2], [11], [12], [13], [14], [15] and the joint embedding of image and question representation [3], [16], [17] have been widely”
VQA With No Questions-Answers Training (2018, background): “As end-to-end training dominates current answering schemes, many works focused on improving the image-question fused features [10, 8, 45], various attention mechanisms for selecting important features [43, 17, 31, 29, 4] and incorporating outputs of other visual tasks [12, 3, 9, 36].”
MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond (2021, background): “Therefore, a local fusion scheme like modulated convolutions is more preferred compared to existing fusion schemes (Fukui et al., 2016; Yu et al., 2017), which are typically applied after visual features are pooled into a single global vector.”
Learning Answer Embeddings for Visual Question Answering (2018, methodology): “In open-end Visual QA, one popular framework of algorithms is to learn a joint image-question embedding and perform multi-way classification (for predicting top-frequency answers) on top [29, 2, 4, 7, 28, 18].”