Deep Convolutional Neural Network Textual Features and Multiple Kernel Learning for Utterance-level Multimodal Sentiment Analysis

By Soujanya Poria, Erik Cambria, Alexander Gelbukh. Conference on Empirical Methods in Natural Language Processing, 2015

475 citations recorded by the source corpus.

We present a novel way of extracting features from short texts, based on the activation values of an inner layer of a deep convolutional neural network. We use the extracted features in multimodal sentiment analysis of short video clips representing one sentence each. We use the combined feature vectors of textual, visual, and audio modalities to train a classifier based on multiple kernel learning, which is known to be good at heterogeneous data. We obtain 14% performance improvement over the state of the art and present a parallelizable decision-level data fusion method, which is much faster, though slightly less accurate.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/0fe86c7fe3028423fea2183ac1a8d5fa8d4a5948

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Multimodal Machine Learning: A Survey and Taxonomy (2017, background): “They have also seen use for multimodal affect recognition [38], [94], [189], multimodal sentiment analysis [169], and multimedia event detection (MED) [245].”
Context-Dependent Sentiment Analysis in User-Generated Videos (2017, methodology): “While the use of knowledge bases was initially more popular for the identiﬁcation of polarity in text (Cambria et al., 2016; Poria et al., 2016c), sentiment analysis researchers have recently been using statistics-based approaches, with a special focus on supervised statistical meth-ods (Socher et a”
A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks (2016, methodology): “Sentiment analysis (Zadeh et al., 2016b) and using multimodal information as a new trend (Zadeh et al., 2016a; Poria et al., 2015a; Tong et al., 2017; Poria et al., 2017; Poria et al., 2016b) is a popular branch of NLP research that aims to understand sentiment of documents automatically using combi”
Multimodal Sentiment Analysis: A Survey of Methods, Trends, and Challenges (2023, methodology): “[135] proposed a multimodal sentiment analysis framework on YouTube videos using multiple kernel learning (MKL) approaches.”
Select-additive learning: Improving generalization in multimodal sentiment analysis (2016, methodology): “We compare the following models: CNN: The state-of-the-art seven layer convolutional neural network architecture used previously for multimodal sentiment analysis [25].”
Speech Intention Classification with Multimodal Deep Learning (2017, methodology): “Finally, we compared our work with some similar research [12–14] (Table 4).”
Multimodal sentiment analysis with word-level fusion and reinforcement learning (2017, background): “…Learning CNN) [36] is a multi-modal SVM-MD (Support Vector Machine Multimodal Dictionary) [46] C-MKL (Convolutional Multiple Kernel Learning) [25] is a multimodal RF (Random Forest) is a baseline intended for comparison to a Random is a baseline which always predicts a random sentiment…”
Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modeling (2018, methodology): “Unlike existing approaches, which use simple concatenation based early fusion [42,45] and non-trainable tensors based fusion [43], this work proposes a hierarchical fusion capable of learning the bimodal and trimodal correlations for data fusion using deep neural networks.”
Tensor Fusion Network for Multimodal Sentiment Analysis (2017, methodology): “C-MKL (Poria et al., 2015) Convolutional MKL-based model is a multimodal sentiment classification model which uses a CNN to extract textual features and uses multiple kernel learning for sentiment analysis.”
The impact of pre- and post-image processing techniques on deep learning frameworks: A comprehensive review for digital pathology image analysis (2020, background): “Several studies demonstrated that aggregating patch-level CNN predictions for WSI classification significantly out-performs patch-level CNNs with major voting [73,109].”
Deep Multimodal Learning: A Survey on Recent Advances and Trends (2017, methodology): “In its simplest form, early fusion involves concatenation of multimodal features as was implemented by Poria et al. [34].”