Learning Factorized Multimodal Representations

By Yao-Hung Hubert Tsai, Percy Liang, Amir Zadeh, Louis-philippe Morency, Ruslan Salakhutdinov. 2018

551 citations recorded by the source corpus.

Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information. Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data: 1) models must learn the complex intra-modal and cross-modal interactions for prediction and 2) models must be robust to unexpected missing or noisy modalities during testing. In this paper, we propose to optimize for a joint generative-discriminative objective across multimodal data and labels. We introduce a model that factorizes representations into two sets of independent factors: multimodal discriminative and modality-specific generative factors. Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks such as sentiment prediction. Modality-specific generative factors are unique for each modality and contain the information required for generating data. Experimental results show that our model is able to learn meaningful multimodal representations that achieve state-of-the-art or competitive performance on six multimodal datasets. Our model demonstrates flexible generative capabilities by conditioning on independent factors and can reconstruct missing modalities without significantly impacting performance. Lastly, we interpret our factorized representations to understand the interactions that influence multimodal learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Multimodal Transformer for Unaligned Multimodal Language Sequences (2019, methodology): “Classical crossmodal alignment, on the other hand, can be expressed as a special (step di-Table (Liang et al., 2018) 38.3 78.4 78.0 0.922 0.681 MFM (Tsai et al., 2019) 36.2 78.1 78.1 0.951 0.662 RAVEN (Wang et al., 2019) 33.2 78.0 76.6 0.915 0.691 MCTN (Pham et al., 2019) 35 (Zadeh et al., 2018b)…”
Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence (2019, unknown): “…multimodal models are creatively used by research communities centered around tasks such as multimodal language analysis [58], sentiment analysis [46, 31], emotion recognition [29], personality traits recognition [56], image captioning [5, 4, 34], multimedia description [50, 51, 62], and video…”
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis (2020, background): “licly available models of [26, 51, 58]).”
Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis (2019, methodology): “Among multi-modal language sentiment or emotion experiments involving unimodal features (Zadeh et al. 2016; Zadeh et al. 2018c; Tsai et al. 2018; Tsai et al. 2019), it is commonly observed that text based features perform better than visual or auditory modes.”
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities (2018, unknown): “Generative-discriminative objectives have been used to learn either joint (Pham et al. 2018; Kiros, Salakhutdinov, and Zemel 2014) or factorized (Tsai et al. 2018) representations.”
Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis (2021, background): “The Multimodal Factorization Model (MFM) (Tsai et al. 2018) learns generative representations to learn the modality-specific generative features along with discriminative representations for classification.”
SMIL: Multimodal Learning with Severely Missing Modality (2021, methodology): “Recently, there have been a wide range of research interests in handling missing modalities for multimodal learning, such as testing-time modality missing (Tsai et al. 2019) and learning with data from unpaired modalities (Shi et al.”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, background): “factors: multimodal discriminative factors for supervised training and intra-modality generative factors for unsupervised training [119].”
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph (2018, background): “Modeling multimodal language has recently become a centric research direction in both NLP and multimodal machine learning (Pham et al., 2018; Tsai et al., 2018; Hazarika et al., 2018; Chen et al., 2017).”
Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors (2018, background): “Currently, the best results are achieved by augmenting this class of models with attention mechanisms (Liang et al. 2018), word-level alignment (Tsai et al. 2018), and more expressive fusion methods (Liu et al. 2018).”
Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models (2019, methodology): “…E B a s e d TELBO (Vedantam et al., 2018) x x x JMVAE (Suzuki et al., 2017) x x MVAE (Wu and Goodman, 2018) x x UNIT (Liu et al., 2017) x x x MFM (Tsai et al., 2019) x x With this in mind, when designing algorithms that imitate the human learning process, we seek a generative model that is able…”
Integrating Multimodal Information in Large Pretrained Transformers (2020, background): “, 2018) and multimodal fusion approaches (Liang et al., 2018; Tsai et al., 2018).”
Multimodal Language Analysis with Recurrent Multistage Fusion (2018, methodology): “We acquire the aligned video and audio features by computing the expectation of their modality feature values over the word utterance time interval (Tsai et al., 2018).”
Think Locally, Act Globally: Federated Learning with Local and Global Representations (2020, background): “Representation Learning involves learning informative features from data that are useful for generative [69, 71, 68] and discriminative [48, 47, 74, 58] tasks.”
CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality (2020, methodology): “In this work, following Zadeh et al. (2017); Liu et al.”