Fusing audio, visual and textual clues for sentiment analysis from multimodal content

By Soujanya Poria, Erik Cambria, N. Howard, G. Huang, Amir Hussain. Neurocomputing, 2016

521 citations recorded by the source corpus.

A huge number of videos are posted every day on social media platforms such as Facebook and YouTube. This makes the Internet an unlimited source of information. In the coming decades, coping with such information and mining useful knowledge from it will be an increasingly difficult task. In this paper, we propose a novel methodology for multimodal sentiment analysis, which consists in harvesting sentiments from Web videos by demonstrating a model that uses audio, visual and textual modalities as sources of information. We used both feature- and decision-level fusion methods to merge affective information extracted from multiple modalities. A thorough comparison with existing works in this area is carried out throughout the paper, which demonstrates the novelty of our approach. Preliminary comparative experiments with the YouTube dataset show that the proposed multimodal system achieves an accuracy of nearly 80%, outperforming all state-of-the-art systems by more than 20%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology

View Paper: https://www.semanticscholar.org/paper/eda501bb1e610098648667eb25273adc4a4dc98d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Experimental and Cognitive Psychology

Citations in the corpus

Ensemble application of convolutional and recurrent neural networks for multi-label text categorization (2017, methodology): “, information retrieval [1], affective computing [2], sentiment analysis [3], email spam detection [4], multimodal content analysis [5], etc.”
Multimodal Sentiment Analysis: A Survey of Methods, Trends, and Challenges (2023, background): “Poria et al. [136] designed a real-time multimodal sentiment understanding model to harvest sentiments from web videos.”
Deep Multimodal Representation Learning: A Survey (2019, background): “For example, by the utilization of fused features from multimodalities, improved performance can be achieved in cross-media analysis tasks, such as video classification [6], event detection [7], [8], and sentiment analysis [9], [10].”
A survey of sentiment analysis in social media (2018, methodology): “Besides, there are novel fusion types such as hybrid multimodal fusion combining both of feature-level fusion and decision-level fusion methods [97, 99], model-level fusion using the correlation between data under different modalities [9, 142], rule-based fusion [28], classification-based fusion usi”
SenticNet 4: A Semantic Resource for Sentiment Analysis Based on Conceptual Primitives (2016, methodology): “To this end, we leverage on VerbNet (Schuler, 2005), the largest English verb lexicon currently available, and Sentic LDA (Poria et al., 2016c), a classiﬁcation framework that integrates commonsense in the calculation of word distributions in the linear discriminant analysis (LDA) algorithm.”
Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions (2022, methodology): “[29] Three modalities 2016 Hybrid Fusion ELM YouTube Explanation of feature extraction as well as model building”
Select-additive learning: Improving generalization in multimodal sentiment analysis (2016, unknown): “A variety of methods have been proposed and extensively discussed in recent years [22, 23, 24].”
Is a picture worth a thousand words? A Deep Multi-Modal Fusion Architecture for Product Classification in e-commerce (2016, methodology): “we experimented with averaging the logits (Table 4, Mean), following (Krizhevsky, Sutskever, and Hinton 2012; Simonyan and Zisserman 2014), and with choosing the network with the maximal confidence (Table 4, Max) following (Poria et al. 2016).”
Deep Convolutional Neural Network Textual Features and Multiple Kernel Learning for Utterance-level Multimodal Sentiment Analysis (2015, background): “While the visual and the audio modalities provide additional evidence that improves classification accuracy, we found the textual modality to have the greater impact on the result (Cambria and Hussain, 2015; Cambria et al., 2013c; Poria et al., 2015a; 2015b).”
Sentiment analysis using deep learning architectures: a review (2019, methodology): “Formore information onmultimodal sentiment analysis, the following popularworks can be referred (Soleymani et al. 2017; Chen et al. 2018; Poria et al. 2016b; Shah et al. 2016; Zhang et al. 2018a; Agarwal et al. 2019).”
Speech Intention Classification with Multimodal Deep Learning (2017, methodology): “Finally, we compared our work with some similar research [12–14] (Table 4).”