MUTAN: Multimodal Tucker Fusion for Visual Question Answering

By Hedi Ben-Younes, Rémi Cadène, Matthieu Cord, Nicolas Thomé. 2017

634 citations recorded by the source corpus.

Bilinear models provide an appealing framework for mixing and merging information in Visual Question Answering (VQA) tasks. They help to learn high level associations between question meaning and visual concepts in the image, but they suffer from huge dimensionality issues.,,We introduce MUTAN, a multimodal tensor-based Tucker decomposition to efficiently parametrize bilinear interactions between visual and textual representations. Additionally to the Tucker framework, we design a low-rank matrix-based decomposition to explicitly constrain the interaction rank. With MUTAN, we control the complexity of the merging scheme while keeping nice interpretable fusion relations. We show how the Tucker decomposition framework generalizes some of the latest VQA architectures, providing state-of-the-art results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Towards a Better Metric for Evaluating Question Generation Systems (2018, methodology): “, 2016) for WikiMovies and (Ben-younes et al., 2017) for VQA.”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “Various approaches to combine spatial image representations and sequential question representations have been proposed [7, 9, 24, 55, 83, 85, 88].”
Attention on Attention for Image Captioning (2019, methodology): “AoA generates an attention gate and an information vector via two linear transformations and applies the gate to the vector to add a second attention, where the techniques are similar to some other work: GLU [10], which replaces RNN and CNN to capture long-range dependencies for language modeling; m”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, methodology): “Moreover, state-of-the-art models such as MUTAN [4] and BAN [20], which are specifically designed for VQA to learn high-level associations between the image and question, get far worse numbers on our dataset.”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, methodology): “Multimodal Tucker Fusion (MUTAN) [6]: This model expresses joint vision-language context in terms of a tensor decomposition, allowing for more expressivity.”
Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers (2020, methodology): “Early method [5] directly adopts CNN features extracted from pre-trained classiﬁcation model as visual representation.”
Modulating early visual processing by language (2017, methodology): “For VQA, we report the results of two state-of-the-art architectures, namely, Multimodal Compact Bilinear pooling network (MCB) [9] (Winner of the VQA challenge 2016) and MUTAN [2].”
Deep Modular Co-Attention Networks for Visual Question Answering (2019, methodology): “[6] exploited different multimodal bilinear pooling methods to integrate the visual features from the image’s spatial grids with the textual features from the questions to predict the attention.”
REMIND Your Neural Network to Prevent Catastrophic Forgetting (2019, methodology): “g with a question string and answer, which can later be used for replay. REMIND can be used with almost any existing VQA system (e.g., attention-based [6,46,78], compositional [7,28], bi-modal fusion [10,19,71]) and it can be applied to similar tasks like image captioning [12] and referring expressi”
Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering (2018, methodology): “Thirdly, better multi-modality fusion approaches, such Bilinear Fusion [9], MCB [7] and MUTAN [4], have been proposed for better capturing the high-level interactions between language and visual features.”
Divide and Conquer: Question-Guided Spatio-Temporal Contextual Attention for Video Question Answering (2020, methodology): “For Image QA, there have been extensive efforts concentrated on (Yang et al. 2016; Xu and Saenko 2016; Wu et al. 2016; Kazemi and Elqursh 2017; Anderson et al. 2018; Gao et al. 2016; Fukui et al. 2016; Ben-Younes et al. 2017) in the last decade.”
Visual Question Generation as Dual Task of Visual Question Answering (2017, methodology): “(2) To reduce the complexity of the full tensor T , Tucker decomposition [3] is introduced as an effective way to factorize T as a tensor product between factor matrices Wq , Wv and Wa, and a core tensor Tc:”
Did the Model Understand the Question? (2018, background): “The Visual Question Answering Task (Agrawal et al., 2015; Teney et al., 2017; Kazemi and Elqursh, 2017; Ben-younes et al., 2017; Zhu et al., 2016) requires a system to answer questions about images (ﬁg.”
TuckER: Tensor Factorization for Knowledge Graph Completion (2019, methodology): “Tucker decomposition, used widely in machine learning (Schein et al., 2016; Ben-Younes et al., 2017; Yang and Hospedales, 2017), factorizes a tensor into a core tensor multiplied by a matrix along each mode.”