MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos

By Amir Zadeh, Rowan Zellers, Eli Pincus, Louis-philippe Morency. arXiv.org, 2016

675 citations recorded by the source corpus.

People are sharing their opinions, stories and reviews through online video sharing websites every day. Studying sentiment and subjectivity in these opinion videos is experiencing a growing attention from academia and industry. While sentiment analysis has been successful for text, it is an understudied research question for videos and multimedia content. The biggest setbacks for studies in this direction are lack of a proper dataset, methodology, baselines and statistical analysis of how information from different modality sources relate to each other. This paper introduces to the scientific community the first opinion-level annotated corpus of sentiment and subjectivity analysis in online videos called Multimodal Opinion-level Sentiment Intensity dataset (MOSI). The dataset is rigorously annotated with labels for subjectivity, sentiment intensity, per-frame and per-opinion annotated visual features, and per-milliseconds annotated audio features. Furthermore, we present baselines for future studies in this direction as well as a new multimodal fusion approach that jointly models spoken words and visual gestures.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/1389564ab24d4b63c921a1ed564e5410b5199f7d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

EgoCom: A Multi-Person Multi-Modal Egocentric Communications Dataset (2020, background): “…speaker detection (Roth et al., 2019), VGG lip reading dataset, an audio-visual dataset for speech recognition and separation (Chung et al., 2017), Mosi, a multimodal corpus of sentiment intensity (Zadeh et al., 2017, 2016), and OpenFace, a multi-modal face recognition (Baltrušaitis et al., 2016).”
A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks (2016, methodology): “Sentiment analysis (Zadeh et al., 2016b) and using multimodal information as a new trend (Zadeh et al., 2016a; Poria et al., 2015a; Tong et al., 2017; Poria et al., 2017; Poria et al., 2016b) is a popular branch of NLP research that aims to understand sentiment of documents automatically using…”
Sentiment analysis using deep learning architectures: a review (2019, background): “• CMU-MOSI dataset (Zadeh et al. 2016) is a popular multimodal dataset which consists of 2199 opinionated utterances from 93 videos on different topics crawled from YouTube.”
Factorized Multimodal Transformer for Multimodal Sequential Learning (2019, methodology): “The results of sentiment analysis experiments on CMU-MOSI dataset are presented in Table 1.”
Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment (2018, background): “MOSI dataset is a multimodal sentiment intensity and subjectivity dataset consisting of 93 reviews with 2199 utterance segments (Zadeh et al., 2016).”
Sentiment Analysis using Imperfect Views from Spoken Language and Acoustic Modalities (2018, methodology): “We present our results and analysis on two datasets, namely, (a) CMU-MOSI (Zadeh et al., 2016) and (b) CMU-MOSEI (Zadeh, 2018a).”
Multimodal sentiment analysis with word-level fusion and reinforcement learning (2017, methodology): “Multimodal models have been used for sentiment analysis [28, 45, 46]; medical purposes, such as detection of suicidal risk, PTSD and depression [29, 34, 35, 43]; emotion recognition [23]; image captioning and media description [8, 41]; question answering [2]; and multimodal translation [31].”
Pengi: An Audio Language Model for Audio Tasks (2023, background): “[63], MELD [48]; Music Analysis: NSynth [19], FMA [9]; Audio Captioning: AudioCaps [32], ClothoV2 [15]; Audio Question and Answering: ClothoAQA [39]; Auxiliary: WavText5K [12], SoundDescs [35], MACS [42], WavCaps [43], FreeSound [20] and FindSound 2 .”
CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality (2020, background): “With the booming of short video applications, nonverbal behaviors (vision and audio) are introduced to solve the above shortcomings (Zadeh et al., 2016; Poria et al., 2017).”
Efficient Low-rank Multimodal Fusion With Modality-Specific Factors (2018, methodology): “We perform our experiments on the following multi-modal datasets, CMU-MOSI (Zadeh et al., 2016a POM (Park et al., 2014b), and IEMOCAP (Busso et al., 2008) for sentiment analysis, speaker traits recognition, and emotion recognition task, where the goal is to identify speakers emotions based on the…”
emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation (2023, methodology): “We tested other speech emotion tasks, including song emotion recognition, emotion prediction in conversation, and sentiment analysis, on RAVDESS-Song (Livingstone and Russo, 2018), IEMOCAP and CMU-MOSI (Zadeh et al., 2016) & CMU-MOSEI (Zadeh et al., 2018).”
SMIL: Multimodal Learning with Severely Missing Modality (2021, background): “• CMU Multimodal Opinion Sentiment Intensity (CMUMOSI) (Zadeh et al. 2016) consists of 2, 199 opinion video clips from YouTube movie reviews.”
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph (2018, methodology): “CMU-MOSEI is the next generation of CMU-MOSI.”
Audio-Visual Fusion for Sentiment Classiﬁcation using Cross-Modal Autoencoder (2018, methodology): “(F1) of 3.7% (3.4) and 2.2% (2.2) on CMU-MOSI, and 3.2% (3.3) and 2.0% (2.1) on CMU-MOSEI datasets, when compared with DCCA and DCCAE, respectively, .”
Integrating Multimodal Information in Large Pretrained Transformers (2020, methodology): “CMU-MOSI (CMU Multimodal Opinion Sentiment Intensity) is a dataset of multimodal language speciﬁcally focused on multimodal sentiment analysis (Zadeh et al., 2016).”
Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions (2022, background): “1 MOSI Dataset MOSI is created by [10].”
Recent trends in deep learning based personality detection (2019, methodology): “This architecture performs better than the state-of-the-art on IEMOCAP [9], MOUD [76] and MOSI [110] datasets and is shown in Fig.”
Tensor Fusion Network for Multimodal Sentiment Analysis (2017, methodology): “Creative and novel applications of using multiple modalities have been among successful recent research directions in machine learning (You et al., 2016; Donahue et al., 2015; Antol et al., 2015; Specia et al., 2016; Tong et al., 2017). opinions from YouTube movie reviews (Zadeh et al., 2016a).”