Tensor Fusion Network for Multimodal Sentiment Analysis
By Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, Louis-philippe Morency. 2017
1,841 citations recorded by the source corpus.
Multimodal sentiment analysis is an increasingly popular research area, which extends the conventional language-based definition of sentiment analysis to a multimodal setup where other relevant modalities accompany language. In this paper, we pose the problem of multimodal sentiment analysis as modeling intra-modality and inter-modality dynamics. We introduce a novel model, termed Tensor Fusion Networks, which learns both such dynamics end-to-end. The proposed approach is tailored for the volatile nature of spoken language in online videos as well as accompanying gestures and voice. In the experiments, our model outperforms state-of-the-art approaches for both multimodal and unimodal sentiment analysis.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology, Multimodal Learning, Sentiment & Opinion Analysis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Soujanya Poria
- Erik Cambria
- Percy Liang
- Louis-philippe Morency
- Devamanyu Hazarika
- Mohit Bansal
- Ruslan Salakhutdinov
- Jianfeng Gao
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Percy Liang
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Louis-philippe Morency
- Jitendra Malik
In Experimental and Cognitive Psychology
- Soujanya Poria
- Erik Cambria
- Navonil Majumder
- Devamanyu Hazarika
- Rada Mihalcea
- Percy Liang
- Louis-philippe Morency
- Amir Zadeh
In Multimodal Learning
- Soujanya Poria
- Louis-philippe Morency
- Percy Liang
- Amir Zadeh
- Devamanyu Hazarika
- Erik Cambria
- Xiaodong He
- Jiele Wu
In Sentiment & Opinion Analysis
- Soujanya Poria
- Percy Liang
- Devamanyu Hazarika
- Amir Zadeh
- Erik Cambria
- Louis-philippe Morency
- Navonil Majumder
- Rada Mihalcea
Citations in the corpus
- Recent Trends in Deep Learning Based Natural Language Processing (2017, methodology): “More specific use cases include applications such as multi-label text categorization [77], multimodal sentiment analysis [78, 79, 80], and subjectivity detection [81].”
- Supervised Multimodal Bitransformers for Classifying Images and Text (2019, methodology): “Related deep fusion methods include multimodal transformers [42], CentralNet [44], MFAS [33] and Tensor Fusion Networks [51].”
- Factorized Multimodal Transformer for Multimodal Sequential Learning (2019, methodology): “…inside a MTL to efficiently extract diverse multimodal Model \ Metric BA F1 MAE Corr MV-LSTM (Rajagopalan et al., 2016) 73.9/– 74.0/– 1.019 0 .601 TFN (Zadeh et al., 2017) 73.9/– 73.4/– 1.040 0 .633 MARN (Zadeh et al., 2018b) 77.1/– 77.0/– 0.968 0.625 MFN (Zadeh et al., 2018a) 77.4/– 77.3/– 0.965…”
- Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities (2018, positive): “The Tensor Fusion Network (Zadeh et al. 2017) and its approximate low-rank model (Liu et al.”
- Harnessing multimodal data integration to advance precision oncology (2021, methodology): “Advances in multimodal ML methods have been impressive in other fields, such as sentiment analysis 86,149–154 , with large benchmark datasets, but the largest multimodal oncological dataset, the TCGA dataset, contains limited data modalities and only a few hundred patients per cancer type.”
- MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis (2020, background): “licly available models of [26, 51, 58]).”
- Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis (2021, methodology): “We can find that the Self-MM gets better results than TFN and LMF and achieve comparable performance with Human-MM.”
- Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions (2022, background): “[32] TFN 2017 Tensor Fusion TFN CMU-MOSI End-to-end learning of intra-modality and inter-modality dynamics 77.”
- Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis (2019, methodology): “An alternative is to use the outer product (Liu et al. 2018; Zadeh et al. 2017) which can represent the interaction between pairs of features, resulting in 2D or 3D arrays that can be processed using advanced methods such as convolutional neural networks (CNNs) (Lawrence et al. 1997).”
- Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence (2019, background): “Tensor Fusion Network (TFN) [55]: Originally proposed for multimodal sentiment analysis, we extend this model for question answering by conditioning the answer based on an outer tensor-product of embeddings of transcript, visual and acoustic modalities.”
- Learning Factorized Multimodal Representations (2018, methodology): “The encoder Q(Zy∣X1∶M) can be parametrized by any model that performs multimodal fusion (Morency et al., 2011; Zadeh et al., 2017).”
- DialogueRNN: An Attentive RNN for Emotion Detection in Conversations (2018, background): “Similar to TFN, this model does not use contextual information.”
- Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment (2018, methodology): “…worth mentioning: (i) our model outperforms the baselines without using the low-level handcrafted acoustic features, indicating the suf-ficiency of MFSCs; (ii) the proposed approach achieves performance comparable to the model us-ing text, audio, and visual data together (Zadeh et al., 2017).”
- Multi-modal Knowledge Graphs for Recommender Systems (2020, background): “For example, it is natural to explore more ways of multi-modal fusion under the framework of multi-modal knowledge graph, such as Tensor Fusion Network (TFN) [34] or Low-rank Multimodal Fusion (LMF) [14].”
- A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks (2016, background): “…2015a; Tong et al., 2017; Poria et al., 2017; Poria et al., 2016b) is a popular branch of NLP research that aims to understand sentiment of documents automatically using combination of various machine learning approaches (Zadeh, 2015; Zadeh et al., 2017b; Poria et al., 2017; Zadeh et al., 2017a).”
- Experts Collaboration Learning for Continual Multi-Modal Reasoning (2023, neutral): “[35], [36], [37], [38], [39], [40], [41], [42], [43] and some multi-modal reasoning benchmarks [11], [30], [44] have been developed.”
- Integrating Multimodal Information in Large Pretrained Transformers (2020, methodology): “LSTMs in TFN, MARN, MFN, RMFN, LFN use latent size of { 16 , 32 , 64 , 128 } .”