Context-Dependent Sentiment Analysis in User-Generated Videos

By Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, Louis-philippe Morency. 2017

874 citations recorded by the source corpus.

Multimodal sentiment analysis is a developing area of research, which involves the identification of sentiments in videos. Current research considers utterances as independent entities, i.e., ignores the interdependencies and relations among the utterances of a video. In this paper, we propose a LSTM-based model that enables utterances to capture contextual information from their surroundings in the same video, thus aiding the classification process. Our method shows 5-10% performance improvement over the state of the art and high robustness to generalizability.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Multimodal Transformer for Unaligned Multimodal Language Sequences (2019, methodology): “Both tasks are frequently used to benchmark models’ ability to fuse multimodal (sentiment) information (Poria et al., 2017; Zadeh et al., 2018a; Liang et al., 2018; Tsai et al., 2019; Pham et al., 2019; Wang et al., 2019).”
Recent Trends in Deep Learning Based Natural Language Processing (2017, methodology): “More specific use cases include applications such as multi-label text categorization [77], multimodal sentiment analysis [78, 79, 80], and subjectivity detection [81].”
A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advances (2022, background): “[383] introduced some variants of the contextual LSTM into the hierarchical architecture for extracting context-dependent multimodal utterance features.”
Learning Factorized Multimodal Representations (2018, neutral): “For a detailed description of the baselines, we point the reader to MFN (Zadeh et al., 2018a), MARN (Zadeh et al., 2018b), TFN (Zadeh et al., 2017), BC-LSTM (Poria et al., 2017), MV-LSTM (Rajagopalan et al., 2016), EF-LSTM (Hochreiter & Schmidhuber, 1997; Graves et al., 2013; Schuster & Paliwal, 199”
Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors (2018, background): “These models are highly modular, and one can build a multimodal model from individual pre-trained uni-modal models and ﬁne-tuning on the output layer (Poria et al. 2017).”
ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection (2018, methodology): “Thus, in our experiments, we extract video-level contextual features for utterances from each modality using the network proposed by Poria et al. 2017b.”
COSMIC: COmmonSense knowledge for eMotion Identification in Conversations (2020, methodology): “Poria et al. (2017) used recurrent neural networks for multimodal emotion recognition followed by (Majumder et al., 2019), where party and global states were used for modeling the emotional dynamics.”
Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances (2019, methodology): “using bc-LSTM architecture [31] that encapsulates the temporal order of the utterances using an LSTM.”
What Makes Multimodal Learning Better than Single (Provably) (2021, methodology): “We follow the data preprocessing method of [33] and obtain 100 dimensions data for audio, 100 dimensions for text, and 500 dimensions for video.”
Recent Trends in Deep Learning Based Natural Language Processing [Review Article] (2018, methodology): “More specific use cases include applications such as multilabel text categorization [76], multimodal sentiment analysis [77]–[79], and subjectivity detection [80].”
Sentiment Analysis using Imperfect Views from Spoken Language and Acoustic Modalities (2018, methodology): “…etc. Similarly, existing and popularly used representations of the acoustic view have generally under-performed compared to the language view (Poria et al., 2017; Zadeh et al., 2018; P´erez-Rosas et al., 2013), indicating that the acoustic view or its representations, by themselves, may not…”
MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation (2021, methodology): “Different models have been proposed to capture the contextual information in previous works, including the LSTM-based model (Poria et al., 2017), the conversational memory network (CMN) model (Hazarika et al., 2018b), interactive conversational memory network (ICON) model (Hazarika et al., 2018a), a”
Deep learning for sentiment analysis: A survey (2018, methodology): “Qian et al. (2015) proposed two more advanced models, tag-guided recursive neural network (TG–RNN), which chooses a composition function according to the part-of-speech tags of a phrase, and tag-embedded recursive neural network/recursive neural tenser network (TE–RNN/RNTN), which learns tag embeddi”
Conversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos (2018, methodology): “Previous research in affective computing has looked at di-alogues as an essential basis to learn emotional dynamics (Sidnell and Stivers, 2012; Poria et al., 2017a; Zhou et al., 2017).”
Knowledge-Enriched Transformer for Emotion Detection in Textual Conversations (2019, methodology): “Similar to (Poria et al., 2017), we clip and pad each utterance Xi j to a fixed m number of tokens.”
Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research (2020, background): “In the literature (Poria et al., 2017), the use of LSTMs to model such sequential sentiment flow has been ineffectual.”
SenticNet 5: Discovering Conceptual Primitives for Sentiment Analysis by Means of Context Embeddings (2018, background): “Such word vectors include more affective clues than regular word vectors and produce better results for tasks such as emotion recognition (Poria et al. 2017), sarcasm detection (Poria et al. 2016) and aspect extraction (Poria, Cambria, and Gelbukh 2016).”