MUREL: Multimodal Relational Reasoning for Visual Question Answering

By Rémi Cadène, Hedi Ben-Younes, Matthieu Cord, Nicolas Thomé. 2019

301 citations recorded by the source corpus.

Multimodal attentional networks are currently state-of-the-art models for Visual Question Answering (VQA) tasks involving real images. Although attention allows to focus on the visual content relevant to the question, this simple mechanism is arguably insufficient to model complex reasoning features required for VQA or other high-level tasks. In this paper, we propose MuRel, a multimodal relational network which is learned end-to-end to reason over real images. Our first contribution is the introduction of the MuRel cell, an atomic reasoning primitive representing interactions between question and image regions by a rich vectorial representation, and modeling region relations with pairwise combinations. Secondly, we incorporate the cell into a full MuRel network, which progressively refines visual and question interactions, and can be leveraged to define visualization schemes finer than mere attention maps. We validate the relevance of our approach with various ablation studies, and show its superiority to attention-based methods on three datasets: VQA 2.0, VQA-CP v2 and TDIUC. Our final MuRel network is competitive to or outperforms state-of-the-art results in this challenging context. Our code is available: github.com/Cadene/murel.bootstrap.pytorch

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, background): “Santoro et al. (2017), Norcliffe-Brown et al. (2018), and Cadene et al. (2019) explore using an attention module to implicitly model the relations between objects in the image.”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “Prominent work includes: (i) advanced attention mechanisms [40, 79, 51]; (ii) better multimodal fusion methods [14, 80, 27, 26]; (iii) multi-step reasoning [77, 22, 15]; (iv) incorporation of object relations [55, 48, 6, 32]; and (v) neural module networks for compositional reasoning [3, 25, 20, 11]”
Learning by Abstraction: The Neural State Machine (2019, methodology): “24.96 HAN [59] 28.65 GVQA [3] 31.30 RAMEN [73] 39.21 BAN [46] 39.31 MuRel [15] 39.54 ReGAT [52] 40.42 NSM 45.80 First, we measure the performance on VQA-CP [3], which provides a new split of the VQA2 dataset [27], where the answer distribution is kept different between the training and the test…”
RUBi: Reducing Unimodal Biases in Visual Question Answering (2019, methodology): “These functions are composed as follows: f ( v i , q i ) = c ( m ( e v ( v i ) , e q ( q i ))) (1) Each one of them can be deﬁned to instantiate most of the state of the art models, such as [26, 41, 19, 42, 17, 43, 16] to cite a few.”
Location-Aware Graph Convolutional Networks for Video Question Answering (2020, methodology): “Some methods (Norcliffe-Brown, Vafeias, and Parisot 2018; Cadène et al. 2019; Li et al. 2019a) took into consideration the object position for image QA tasks.”
Coarse-to-Fine Reasoning for Visual Question Answering (2021, methodology): “, through message passing [50], pairwise relationship modeling [4], adversarial learning [8, 31, 51], or graph parsing methods defined by inter/intra-class edges [15].”
Violin: A Large-Scale Dataset for Video-and-Language Inference (2020, unknown): “Many neural network models have been proposed for these tasks, such as more advanced attention mechanisms [64, 43, 70], better multimodal fusion methods [15, 71, 31, 30], the use of multi-step reasoning [24, 17, 7], the incorporation of relations [49, 38, 45], and neural module networks for composit”
LPF: A Language-Prior Feedback Objective Function for De-biased Visual Question Answering (2021, background): “Nowadays, the state-of-the-art VQA models [3, 7, 17, 20, 27, 29, 32] can achieve well performance on the”
CogVLM: Visual Expert for Pretrained Language Models (2023, unknown): “The experimental results on TDIUC compare our model against the specialist SOTA method MUREL [Cadene et al., 2019] are shown in Figure 4.”
Reasoning with Heterogeneous Graph Alignment for Video Question Answering (2020, background): “• ST-VQA (cid:5) is the current state-of-the-art on TGIF-QA dataset (Jang et al. 2019).”
An Empirical Study on Leveraging Scene Graphs for Visual Question Answering (2019, background): “Recent works on Visual QA [7, 11, 43, 46, 59, 70] and visual reasoning [33, 59] have also begun to exploit scene graphs.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “Further, a multimodal relational network (MuRel) (Cadene et al., 2019) is proposed to learn end-to-end reasoning over real images.”
Structured Multimodal Attentions for TextVQA (2020, background): “MUREL [30] goes one step further to model spatial-semantic pairwise relations between all pairs of regions for relation reasoning, in addition to a rich vectorial representation for interaction between region’s visual content and question.”
Cascade Reasoning Network for Text-based Visual Question Answering (2020, background): “Relational reasoning has been widely explored in computer vision tasks [15], from region classification [9] to visual question answering [8, 48, 49], and further promoted after graph neural networks appeared [36].”
VQA With No Questions-Answers Training (2018, background): “Many recent works focused on improving the image-question fused features [17, 9, 84, 10, 16], attention mechanisms for selecting important features [80, 54, 47, 4, 12, 59, 50, 27], including self and guided attention [83, 21, 82], applying pre-trained networks [65, 46, 86], and incorporating outputs”
A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2021, background): “Currently, the most common existing methods use attention [125,127,129] and memory [126] mechanisms to efficiently learn visual artifacts and the semantic correlations that allow questions to be answered accurately.”