Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modeling
By Navonil Majumder, Devamanyu Hazarika, Alexander Gelbukh, Erik Cambria, Soujanya Poria. Knowledge-Based Systems, 2018
380 citations recorded by the source corpus.
Multimodal sentiment analysis is a very actively growing field of research. A promising area of opportunity in this field is to improve the multimodal fusion mechanism. We present a novel feature fusion strategy that proceeds in a hierarchical fashion, first fusing the modalities two in two and only then fusing all three modalities. On multimodal sentiment analysis of individual utterances, our strategy outperforms conventional concatenation of features by 1%, which amounts to 5% reduction in error rate. On utterance-level multimodal sentiment analysis of multi-utterance video clips, for which current state-of-the-art techniques incorporate contextual information from other utterances of the same clip, our hierarchical fusion gives up to 2.4% (almost 10% error rate reduction) over currently used concatenation. The implementation of our method is publicly available in the form of open-source code.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology
View Paper: https://www.semanticscholar.org/paper/0b5aef2894d3248fb5ecc955d50501f0aa276036
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Soujanya Poria
- Erik Cambria
- Devamanyu Hazarika
- Navonil Majumder
- Rada Mihalcea
- Ringki Das
- Thoudam Doren Singh
- Amir Hussain
In Computer Vision and Pattern Recognition
- Soujanya Poria
- Erik Cambria
- Devamanyu Hazarika
- Rada Mihalcea
- Navonil Majumder
- Ringki Das
- Thoudam Doren Singh
- Amir Hussain
In Experimental and Cognitive Psychology
- Soujanya Poria
- Navonil Majumder
- Erik Cambria
- Devamanyu Hazarika
- Rada Mihalcea
- Amir Hussain
- Ringki Das
- Thoudam Doren Singh
Citations in the corpus
- Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions (2022, methodology): “[36] CHFusion 2018 Hierarchical Feature Fusion CNN CMU-MOSI Hierarchical Fusion, in which the two modalities are fused first, followed by the three modalities.”
- A survey on sentiment analysis methods, applications, and challenges (2022, methodology): “In the work of Majumder et al. (2018) and Poria et al. (2018b) feature fusion technique that is hierarchical in nature, merging the two modalities first and subsequently all three modalities.”
- MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations (2018, background): “Although significant research work has been carried out on multimodal emotion recognition using audio, visual, and text modalities (Poria et al., 2017a; Zadeh et al., 2016a; Wollmer et al., 2013; Majumder et al., 2018), much less work has been devoted to emotion recognition in conversations (ERC).”
- Multimodal Sentiment Analysis: A Survey of Methods, Trends, and Challenges (2023, methodology): “[139] addressed the multimodal sentiment analysis framework using a contextaware hierarchical fusion scheme.”