Gated Multimodal Units for Information Fusion

By John Arevalo, Thamar Solorio, Manuel Montes-y-Gómez, Fabio A. González. 2017

625 citations recorded by the source corpus.

This paper presents a novel model for multimodal learning based on gated neural networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit using multiplicative gates. It was evaluated on a multilabel scenario for genre classification of movies using the plot and the poster. The GMU improved the macro f-score performance of single-modality approaches and outperformed other fusion strategies, including mixture of experts models. Along with this work, the MM-IMDb dataset is released which, to the best of our knowledge, is the largest publicly available multimodal dataset for genre prediction on movies.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Multimodal Learning

Citations in the corpus

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, methodology): “Other datasets include grounded entailment, which exploits the fact that one of the large-scale natural language inference datasets was constructed using captions as premises, yielding a image, premise, hypothesis triplet with associated entailment label [79]; as well as MM-IMDB, where the aim is to”
Supervised Multimodal Bitransformers for Classifying Images and Text (2019, methodology): “…text-dominated multimodal classiﬁcation, and evaluate on the following three tasks, which have been used in the past speciﬁcally for evaluating multimodal classiﬁcation architectures (Kiela et al. 2018): MM-IMDB (Arevalo et al. 2017), Food101 (Wang et al. 2015) and V-SNLI (Vu et al. 2018).”
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition (2021, background): “Multi-modal learning has been studied from multiple perspectives, such as two stream networks that fuse decisions from multiple modalities for classification [41, 7, 26, 27, 3], and cross-modal learning that takes one modality as input and make prediction on the other modality [29, 2, 62, 1, 15, 42]”
EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition (2019, background): “(iii) Gating fusion was introduced in [4], where a gate neuron takes as input the features from all modalities to learn the importance of one modality w.”
Towards Accurate Scene Text Recognition With Semantic Reasoning Networks (2020, methodology): “Inspired by the gated unit [1], we introduce some trainable weights to balance the contribution of features from different domains in our VSFD.”
SMIL: Multimodal Learning with Severely Missing Modality (2021, methodology): “• The Multimodal IMDb (MM-IMDb) (Arevalo et al. 2017) contains two modalities: image and text.”
Integrating Scene Text and Visual Appearance for Fine-Grained Image Classification (2017, methodology): “Although how to fuse the multi-modal information is not the main focus of this paper, we tried several fusion approaches, like concatenation, linear sum and GMU [85].”
Audiovisual SlowFast Networks for Video Recognition (2020, background): “Beyond audio and visual modalities, extensive research has been conducted in various other instantiations of multi-modal learning, including vision and motion [62, 16, 77, 9], vision and language [4, 18], and learning from physiological data [44].”
Illustrative Language Understanding: Large-Scale Visual Grounding with Image Search (2018, background): “Similar gating mechanisms can be found in LSTMs (Hochreiter and Schmidhu-ber, 1997) and other multimodal models (Arevalo et al., 2017; Wang et al., 2018; Kiela et al., 2018).”
Are we pretraining it right? Digging deeper into visio-linguistic pretraining (2020, methodology): “To study this, we look at results on two small downstream datasets, VizWiz [18] and MM-IMDB [5] from Section 3.1.”
Audio-Visual Event Localization in Unconstrained Videos (2018, neutral): “If there is no audio-visual event in a frame, random background regions that do not focus on objects will be attended (see 2rd frame of ﬁrst example and 5th frame of the second example); otherwise, the attention will focus on sounding sources Unit (GMU) in [56], Concatenation (Concat), and MRN [47].”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, methodology): “unit is a method that can be viewed as assigning attention weights to images and text based on gating [164].”
Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis (2019, methodology): “To decrease the impact of noisy unimodal features during multimodal training, before the Kronecker Product, we employed a gating-based attention mechanism that controls the expressiveness of features of each modality [60].”
What Makes Training Multi-Modal Classification Networks Hard? (2019, background): “Our work is related to previous research on multi-modal networks [7] for classiﬁcations [41, 49, 19, 21, 12, 6, 10, 31], which primarily uses pre-training in contrast to our joint training.”