MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis

By Devamanyu Hazarika, Roger Zimmermann, Soujanya Poria. ACM Multimedia, 2020

1,255 citations recorded by the source corpus.

Multimodal Sentiment Analysis is an active area of research that leverages multimodal signals for affective understanding of user-generated videos. The predominant approach, addressing this task, has been to develop sophisticated fusion techniques. However, the heterogeneous nature of the signals creates distributional modality gaps that pose significant challenges. In this paper, we aim to learn effective modality representations to aid the process of fusion. We propose a novel framework, MISA, which projects each modality to two distinct subspaces. The first subspace is modality-invariant, where the representations across modalities learn their commonalities and reduce the modality gap. The second subspace is modality-specific, which is private to each modality and captures their characteristic features. These representations provide a holistic view of the multimodal data, which is used for fusion that leads to task predictions. Our experiments on popular sentiment analysis benchmarks, MOSI and MOSEI, demonstrate significant gains over state-of-the-art models. We also consider the task of Multimodal Humor Detection and experiment on the recently proposed UR_FUNNY dataset. Here too, our model fares better than strong baselines, establishing MISA as a useful multimodal framework.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Social Psychology

View Paper: https://www.semanticscholar.org/paper/dde6151c4e82cbfd573e3ec1d1a36af4a1121220

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Soujanya Poria
2. Erik Cambria
3. Amir Hussain
4. Kinjal U. Adhvaryu
5. Ankita Gandhi
6. Devamanyu Hazarika
7. Hua Xu
8. Jiele Wu

In Computer Vision and Pattern Recognition

1. Soujanya Poria
2. Erik Cambria
3. Amir Hussain
4. Kinjal U. Adhvaryu
5. Ankita Gandhi
6. Hua Xu
7. Jiele Wu
8. Wenmeng Yu

In Social Psychology

1. Erik Cambria
2. Soujanya Poria
3. Amir Hussain
4. Devamanyu Hazarika
5. Navonil Majumder
6. Rada Mihalcea
7. Louis-philippe Morency
8. Amir Zadeh

Citations in the corpus

• Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions (2022, methodology): “It makes use of the MOSI and MOSEI datasets[91].”
• Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis (2021, methodology): “The first is to use the pre-trained BERT as a language feature extraction module (Hazarika, Zimmermann, and Poria 2020).”