Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding

By Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, Marcus Rohrbach. 2016

1,594 citations recorded by the source corpus.

Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years. However, tasks such as visual question answering require combining these vector representations with each other. Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations. We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors. As the outer product is typically infeasible due to its high dimensionality, we instead propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and expressively combine multimodal features. We extensively evaluate MCB on the visual question answering and grounding tasks. We consistently show the benefit of MCB over ablations without MCB. For visual question answering, we present an architecture which uses MCB twice, once for predicting attention over spatial features and again to combine the attended representation with the question representation. This model outperforms the state-of-the-art on the Visual7W dataset and the VQA challenge.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Learned in Translation: Contextualized Word Vectors (2017, methodology): “For VQA, Fukui et al. [2016] propose to combine text representations with visual representations extracted by a pretrained residual network [He et al., 2016].”
Bidirectional Attention Flow for Machine Comprehension (2016, methodology): “Several approaches to constructing the attention matrix have been used including element-wise product, element-wise sum, concatenation and Multimodal Compact Bilinear Pooling (Fukui et al., 2016).”
UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “For example, MCB (Fukui et al., 2017) and BAN (Kim et al., 2018) proposed advanced multimodal fusion methods for Visual Question Answering (VQA) (Antol et al., 2015).”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “Prominent work includes: (i) advanced attention mechanisms [40, 79, 51]; (ii) better multimodal fusion methods [14, 80, 27, 26]; (iii) multi-step reasoning [77, 22, 15]; (iv) incorporation of object relations [55, 48, 6, 32]; and (v) neural module networks for compositional reasoning [3, 25, 20, 11]”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “These applications include automatic image and video captioning [20, 36, 60, 61], visual question answering [9, 29, 51, 63], visual content retrieval based on textual queries [32, 56, 63], temporal localization of events in videos using natural language [15, 26] or video summarization with natural l”
Towards VQA Models That Can Read (2019, methodology): “VQA models typically use some variant of attention to get a representation of the image that is relevant for answering the question [2, 7, 30, 47, 48, 51, 17].”
ShapeWorld - A new test methodology for multimodal language understanding (2017, methodology): “…VQA setup (i.e., yes/no questions), it neither requires the evaluated model to generate answers nor to rephrase the problem to ﬁt it into a classiﬁcation task of some sort – for instance, over the 1000/3000 most common answers, as is common practice recently (Lu et al., 2016; Fukui et al., 2016).”
How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, neutral): “Various methods have been introduced (Fukui et al., 2016; Yang et al., 2016; Anderson et al., 2018a; Jiang et al., 2018; Gao et al., 2019; Jiang et al., 2020).”
Achieving Human Parity on Visual Question Answering (2021, background): “Traditional cross-modal fusionmethods gradually improve from a simple linear model [1] to bilinear pooling ones [20, 94], and attention-based fusion ones [52, 89], where questions and image features are fully interactedwith each other.”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “Various approaches to combine spatial image representations and sequential question representations have been proposed [7, 10, 25, 57, 86, 88, 91].”
Attention on Attention for Image Captioning (2019, methodology): “AoA generates an attention gate and an information vector via two linear transformations and applies the gate to the vector to add a second attention, where the techniques are similar to some other work: GLU [10], which replaces RNN and CNN to capture long-range dependencies for language modeling; m”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, background): “Attention-based models [11, 25, 39, 40, 41, 47] better guide the model in answering the questions by highlighting image regions that are relevant to the question.”
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, methodology): “Table 2: Accuracy breakdown over different answer types achieved by MCB [9] and HieCoAtt [25] models when trained/tested on unbalanced/balanced VQA datasets.”
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering (2017, methodology): “Typically, these models can be characterized as top-down approaches, with context provided by a representation of a partially-completed caption in the case of image captioning [33, 27, 47, 45], or a representation of the question in the case of VQA [11, 28, 44, 46, 49].”
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering (2019, background): “popular across the board, with a host of datasets being constructed [3, 9, 13, 36, 17] and numerous models being proposed [4, 33, 5, 8, 10].”
Machine Comprehension Using Match-LSTM and Answer Pointer (2016, background): “The pointer mechanism has inspired some recent work on language processing (Gu et al., 2016; Kadlec et al., 2016).”
Bilinear Attention Networks (2018, methodology): “networks (MRN) to efficiently utilize the multiple bilinear attention maps of the BAN, unlike the previous works [6, 15] where multiple attention maps are used by concatenating the attended features.”