Multimodal sentiment analysis with word-level fusion and reinforcement learning

By Minghai Chen, Sen Wang, Percy Liang, Tadas Baltrusaitis, Amir Zadeh, Louis-philippe Morency. International Conference on Multimodal Interaction, 2017

312 citations recorded by the source corpus.

With the increasing popularity of video sharing websites such as YouTube and Facebook, multimodal sentiment analysis has received increasing attention from the scientific community. Contrary to previous works in multimodal sentiment analysis which focus on holistic information in speech segments such as bag of words representations and average facial expression intensity, we propose a novel deep architecture for multimodal sentiment analysis that is able to perform modality fusion at the word level. In this paper, we propose the Gated Multimodal Embedding LSTM with Temporal Attention (GME-LSTM(A)) model that is composed of 2 modules. The Gated Multimodal Embedding allows us to alleviate the difficulties of fusion when there are noisy modalities. The LSTM with Temporal Attention can perform word level fusion at a finer fusion resolution between the input modalities and attends to the most important time steps. As a result, the GME-LSTM(A) is able to better model the multimodal structure of speech through time and perform better sentiment comprehension. We demonstrate the effectiveness of this approach on the publicly-available Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis (CMU-MOSI) dataset by achieving state-of-the-art sentiment classification and regression results. Qualitative analysis on our model emphasizes the importance of the Temporal Attention Layer in sentiment prediction because the additional acoustic and visual modalities are noisy. We also demonstrate the effectiveness of the Gated Multimodal Embedding in selectively filtering these noisy modalities out. These results and analysis open new areas in the study of sentiment analysis in human communication and provide new models for multimodal fusion.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology

View Paper: https://www.semanticscholar.org/paper/81980911e5d7c4776962c0f6ae3fe58b5ef07b80

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Experimental and Cognitive Psychology

Citations in the corpus

Deep Multimodal Representation Learning: A Survey (2019, background): “[93] proposed to train a feature selection module used for determining whether an input at time step t should be included or not during encoding, such that salient features can be involved while noise will be excluded.”
Learning Factorized Multimodal Representations (2018, methodology): “, 2018) and better sample generation quality than its counterparts (Chen et al., 2016; Higgins et al., 2016; Kingma & Welling, 2013). However, WAEs are designed for unimodal data and do not consider factorized distributions over latent variables that generate multimodal data. Therefore, we propose a”
Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis (2019, background): “In their work (Chen et al. 2017) pro-pose improvements to multi-modal embeddings using reinforcement learning to align the multi-modal embedding at the word level by removing noises.”
Integrating Multimodal Information in Large Pretrained Transformers (2020, methodology): “For each word, we align all three modalities following the convention established in (Chen et al., 2017).”
Deep Facial Expression Recognition: A Survey (2018, methodology): “For instance, [219] used word-level modality fusion to align each word to corresponding video frames and audio segments.”
ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection (2018, background): “Numerous previous approaches have relied on fusion techniques that leverage multiple modalities for affect recognition (Soleymani et al., 2012; Zadeh et al., 2017; Chen et al., 2017; Tzirakis et al., 2017; Zadeh et al., 2018b).”
COSMIC: COmmonSense knowledge for eMotion Identification in Conversations (2020, background): “Some of the other important works include (Hazarika et al., 2018a,b; Zadeh et al., 2018b; Chen et al., 2017; Zadeh et al., 2018a).”
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis (2020, unknown): “Approaches includemultimodal-awareword embeddings [56], recurrent multi-stage fusion [24], graph-based fusion [30, 60], recurrent networks (RNNs), attention-models, memory mechanisms, and transformer-based models [8, 46, 46, 52, 56, 59–61].”
DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation (2019, background): “Some of the important works include (Poria et al., 2017; Chen et al., 2017; Zadeh et al., 2018a,b; Hazarika et al., 2018a,b), where mainly deep learning-based techniques have been employed for emotion (and sentiment) recognition in conversation, in only textual and multimodal settings.”
Factorized Multimodal Transformer for Multimodal Sequential Learning (2019, methodology): “After resampling using a reference clock, modalities can follow the same frequency (Chen et al., 2017).”
Efficient Low-rank Multimodal Fusion With Modality-Specific Factors (2018, methodology): “We calculate the visual and acoustic features by taking the average of their feature values over the word time interval (Chen et al., 2017).”
Sentiment analysis using deep learning architectures: a review (2019, methodology): “(2017c) Temporal attention + LSTM improves the sentiment prediction despite the presence of noisy acoustic and visual modalities (Chen 2017) Hierarchical attention can model the aspect information of the text, which further helps in capturing the sentiment information (Cheng et al.”
Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment (2018, methodology): “LSTM(A) introduces a word-level LSTM with temporal attention structure to predict sentiments on MOSI dataset (Chen et al., 2017).”
Sentiment Analysis using Imperfect Views from Spoken Language and Acoustic Modalities (2018, background): “…from these individual different views add richness to the sentiment classiﬁcation, the intra and inter view-interactions play an important role in better sentiment classiﬁcation (Zadeh et al., 2017; Chen et al., 2018; Rajagopalan et al., 2016; Nojavanasghari et al., 2016; Xu et al., 2013).”
DialogueRNN: An Attentive RNN for Emotion Detection in Conversations (2018, methodology): “(2017) successfully used RNN-based deep networks for multimodal emotion recognition, which was followed by other works (Chen et al. 2017; Zadeh et al. 2018a; 2018b).”
Multimodal Language Analysis with Recurrent Multistage Fusion (2018, background): “…analysis and has recently become a centric research direction in multimodal natural language processing (Liu et al., 2018; Pham et al., 2018; Chen et al., 2017), multimodal speech recognition (Sun et al., 2016; Gupta et al., 2017; Harwath and Glass, 2017; Kamper et al., 2017), as well as…”
Sentiment Analysis Based on Deep Learning: A Comparative Study (2020, background): “eving comparable performances with fewer parameters on sentiment analysis tasks Electronics 2020, 9, 483 13 of 29 Table 1. Cont. No. Year Study Research Work Method Dataset Target 27 2017 Chen et al. [65] Multimodal sentiment analysis with word-level fusion and reinforcement learning Gated multimoda”