Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis

By Soujanya Poria, Iti Chaturvedi, Erik Cambria, Amir Hussain. Industrial Conference on Data Mining, 2016

562 citations recorded by the source corpus.

Technology has enabled anyone with an Internet connection to easily create and share their ideas, opinions and content with millions of other people around the world. Much of the content being posted and consumed online is multimodal. With billions of phones, tablets and PCs shipping today with built-in cameras and a host of new video-equipped wearables like Google Glass on the horizon, the amount of video on the Internet will only continue to increase. It has become increasingly difficult for researchers to keep up with this deluge of multimodal content, let alone organize or make sense of it. Mining useful knowledge from video is a critical need that will grow exponentially, in pace with the global growth of content. This is particularly important in sentiment analysis, as both service and product reviews are gradually shifting from unimodal to multimodal. We present a novel method to extract features from visual and textual modalities using deep convolutional neural networks. By feeding such features to a multiple kernel learning classifier, we significantly outperform the state of the art of multimodal emotion recognition and sentiment analysis on different datasets.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology

View Paper: https://www.semanticscholar.org/paper/0d21cc2677e544b46673ff19ad4f378f32129069

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Experimental and Cognitive Psychology

Citations in the corpus

Sentiment Analysis Is a Big Suitcase (2017, unknown): “18”
SMIL: Multimodal Learning with Severely Missing Modality (2021, methodology): “Early fusion is a common method which fuses different modalities by feature concatenation, and it has been widely adopted in previous studies (Wang et al. 2017; Poria et al. 2016).”
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph (2018, unknown): “We also compare to the following baseline models: † BC-LSTM (Poria et al., 2017), ♣ C-MKL (Poria et al., 2016), ♭ DF (Nojavanasghari et al., 2016), ♡ SVM (Cortes and Vapnik, 1995; Zadeh et al., 2016b; Perez-Rosas et al., 2013; Park et al., 2014), ● RF (Breiman, 2001), THMM (Morency et al., 2011), SA”
A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks (2016, background): “…context-aware paradigms such as convolutional networks (Poria et al., 2016a), recurrent belief networks (Chaturvedi et al., 2016), statistical learning theory (Oneto et al., 2016), convolutional multiple kernel learning (Poria et al., 2016b), and commonsense reasoning (Cambria and Hussain, 2015).”
Multimodal Language Analysis with Recurrent Multistage Fusion (2018, methodology): “Conventional supervised learning methods are utilized to discover intra-modal and cross-modal interactions without specific model design (Wang et al., 2016; Poria et al., 2016).”
Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment (2018, methodology): “Multiple kernel learning is used as the final classifier (Poria et al., 2016).”
Efficient Low-rank Multimodal Fusion With Modality-Specific Factors (2018, methodology): “Several works that use this method of fusion (Poria et al., 2016) , (Wang et al.”
How to get pavement distress detection ready for deep learning? A systematic approach (2017, methodology): “In future work, we will also evaluate more advanced network structures like ResNets [34], [35], latest InceptionNets [33], [36], and sequential classifiers such as stack CNNRNN [37], [38].”
Context-Dependent Sentiment Analysis in User-Generated Videos (2017, methodology): “While the use of knowledge bases was initially more popular for the identiﬁcation of polarity in text (Cambria et al., 2016; Poria et al., 2016c), sentiment analysis researchers have recently been using statistics-based approaches, with a special focus on supervised statistical meth-ods (Socher et…”
Speech Intention Classification with Multimodal Deep Learning (2017, methodology): “A ConvNet structure was used as a trainable textual feature extractor [13, 14], and another ConvNet structure extracted the visual features [14].”
CARER: Contextualized Affect Representations for Emotion Recognition (2018, background): “Feature Representations: Recent emotion recognition systems employ representation learning for automatic feature extraction (Poria et al., 2016; Savigny and Purwarianti, 2017; Abdul-Mageed and Ungar, 2017).”