Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

By Amir Zadeh, Percy Liang, Soujanya Poria, Erik Cambria, Louis-philippe Morency. 2018

1,662 citations recorded by the source corpus.

Analyzing human multimodal language is an emerging area of research in NLP. Intrinsically this language is multimodal (heterogeneous), sequential and asynchronous; it consists of the language (words), visual (expressions) and acoustic (paralinguistic) modalities all in the form of asynchronous coordinated sequences. From a resource perspective, there is a genuine need for large scale datasets that allow for in-depth studies of this form of language. In this paper we introduce CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI), the largest dataset of sentiment analysis and emotion recognition to date. Using data from CMU-MOSEI and a novel multimodal fusion technique called the Dynamic Fusion Graph (DFG), we conduct experimentation to exploit how modalities interact with each other in human multimodal language. Unlike previously proposed fusion techniques, DFG is highly interpretable and achieves competative performance when compared to the previous state of the art.

Subfields: Artificial Intelligence, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Multimodal Transformer for Unaligned Multimodal Language Sequences (2019, methodology): “Both tasks are frequently used to benchmark models’ ability to fuse multimodal (sentiment) information (Poria et al., 2017; Zadeh et al., 2018a; Liang et al., 2018; Tsai et al., 2019; Pham et al., 2019; Wang et al., 2019).”
Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders (2019, methodology): “To demonstrate domain invariant transferability of the proposed representation across different datasets, the Mocking-jay model is pre-trained on LibriSpeech and applied on the MOSEI [25] dataset.”
Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions (2022, background): “[47] DFG 2018 Word-level Fusion MFN+DFG CMU-MOSEI Multi-view sequential learning that consists of three main components System of LSTMs consists of multiple Longshort Term Memory (LSTM) networks, Deltamemory Attention Network 85.”
Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis (2021, methodology): “MOSEI.”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “Researchers also evaluate representations for sentiment analysis with the INTERFACE [233] and MOSEI (CMU Multimodal Opinion Sentiment and Emotion Intensity) [234] datasets.”
Emotions Don't Lie: An Audio-Visual Deepfake Detection Method using Affective Cues (2020, methodology): “There are many other applications of video processing that use and combine multiple modalities for audio-visual speech recognition [28], emotion recognition [40, 63], and language and vision tasks [17, 29].”
MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations (2018, background): “The other publicly available multimodal emotion and sentiment recognition datasets are MOSEI (Zadeh et al., 2018b), MOSI (Zadeh et al., 2016b), and MOUD (P´erez-Rosas et al., 2013), however none of these datasets is conversational.”
Factorized Multimodal Transformer for Multimodal Sequential Learning (2019, methodology): “Multimodal models based on LSTMs include Multi-view LSTMs Rajagopalan et al. (2016), Memory Fusion Network (Zadeh et al., 2018a) with its recurrent and graph variants (Liang et al., 2018; Zadeh et al., 2018c), as well as Multi-attention Recurrent Networks (Zadeh et al., 2018b).”
Exploring Deep Multimodal Fusion of Text and Photo for Hate Speech Classification (2019, methodology): “Attention mechanism was initially proposed in neural machine translation to dynamically adjust the focus on the source sentence (Bahdanau et al., 2014), but its application has been extended to many areas including multimodal fusion (Lu et al., 2018; Ghosal et al., 2018; Bagher Zadeh et al., 2018).”
Integrating Multimodal Information in Large Pretrained Transformers (2020, methodology): “We use two evaluation metrics for BA and F1, one used in (Zadeh et al., 2018d) and one used in (Tsai et al.”
i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data (2023, methodology): “We gather data from CMU Multimodal Opinion Sentiment and Emotion Intensity (CMUMOSEI) (Zadeh et al., 2018) and Spoken Language Understanding Evaluation (SLUE) (Shon et al., 2022).”
emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation (2023, methodology): “We tested other speech emotion tasks, including song emotion recognition, emotion prediction in conversation, and sentiment analysis, on RAVDESS-Song (Livingstone and Russo, 2018), IEMOCAP and CMU-MOSI (Zadeh et al., 2016) & CMU-MOSEI (Zadeh et al., 2018).”
Pengi: An Audio Language Model for Audio Tasks (2023, background): “[63], MELD [48]; Music Analysis: NSynth [19], FMA [9]; Audio Captioning: AudioCaps [32], ClothoV2 [15]; Audio Question and Answering: ClothoAQA [39]; Auxiliary: WavText5K [12], SoundDescs [35], MACS [42], WavCaps [43], FreeSound [20] and FindSound 2 .”
Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis (2019, methodology): “Among multi-modal language sentiment or emotion experiments involving unimodal features (Zadeh et al. 2016; Zadeh et al. 2018c; Tsai et al. 2018; Tsai et al. 2019), it is commonly observed that text based features perform better than visual or auditory modes.”
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis (2020, methodology): “com/platform/ 4Please refer to [19, 60] and their respective SDKs (https://github.”