Select-additive learning: Improving generalization in multimodal sentiment analysis
By Haohan Wang, Aaksha Meghawat, Louis-philippe Morency, Eric P. Xing. 2016
176 citations recorded by the source corpus.
Multimodal sentiment analysis is drawing an increasing amount of attention these days. It enables mining of opinions in video reviews which are now available aplenty on online platforms. However, multimodal sentiment analysis has only a few high-quality data sets annotated for training machine learning algorithms. These limited resources restrict the generalizability of models, where, for example, the unique characteristics of a few speakers (e.g., wearing glasses) may become a confounding factor for the sentiment classification task. In this paper, we propose a Select-Additive Learning (SAL) procedure that improves the generalizability of trained neural networks for multimodal sentiment analysis. In our experiments, we show that our SAL approach improves prediction accuracy significantly in all three modalities (verbal, acoustic, visual), as well as in their fusion. Our results show that SAL, even when trained on one dataset, achieves good generalization across two new test datasets.
Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, Natural Language Processing, Multimodal Learning, Sentiment & Opinion Analysis, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Soujanya Poria
- Erik Cambria
- Percy Liang
- Louis-philippe Morency
- Devamanyu Hazarika
- Amir Zadeh
- Eric P. Xing
- Navonil Majumder
In Experimental and Cognitive Psychology
- Soujanya Poria
- Percy Liang
- Louis-philippe Morency
- Amir Zadeh
- Devamanyu Hazarika
- Erik Cambria
- Navonil Majumder
- Roger Zimmermann
In Natural Language Processing
- Soujanya Poria
- Erik Cambria
- Louis-philippe Morency
- Percy Liang
- Amir Zadeh
- Devamanyu Hazarika
- Eric P. Xing
- Pengcheng Yin
In Multimodal Learning
- Soujanya Poria
- Percy Liang
- Louis-philippe Morency
- Amir Zadeh
- Devamanyu Hazarika
- Erik Cambria
- Roger Zimmermann
- Liu Ziyin
In Sentiment & Opinion Analysis
- Percy Liang
- Soujanya Poria
- Amir Zadeh
- Louis-philippe Morency
- Devamanyu Hazarika
- Erik Cambria
- Ringki Das
- Thoudam Doren Singh
In Signal Processing
- Amir Zadeh
- Louis-philippe Morency
- Soujanya Poria
- Percy Liang
- Devamanyu Hazarika
- Erik Cambria
- Ringki Das
- Thoudam Doren Singh
Citations in the corpus
- Learning Robust Global Representations by Penalizing Local Predictive Power (2019, methodology): “More recent DG approaches incorporate adversarial (or similar) techniques (Ghifary et al., 2015; Wang et al., 2016; Motiian et al., 2017b; Li et al., 2018a; Carlucci et al., 2018), or build ensembles of per-domain models that are then fused representations together (Bousmalis et al.”
- Learning Robust Representations by Projecting Superficial Statistics Out (2018, background): “…into two categories: reducing the inter-domain differences of representations through adversarial (or similar) techniques (Ghifary et al., 2015; Wang et al., 2016; Motiian et al., 2017; Li et al., 2018; Carlucci et al., 2018), or building an ensemble of one-for-each-domain deep models and then…”
- Removing Confounding Factors Associated Weights in Deep Neural Networks Improves the Prediction Accuracy for Healthcare Applications (2018, methodology): “12,25 However, our method does not require full correspondence between X , y , and s .”
- High-Frequency Component Helps Explain the Generalization of Convolutional Neural Networks (2019, background): “In this paper, inspired by previous understandings that convolutional neural networks (CNN) can learn from confounding signals [59] and superficial signals [29, 19, 58], we investigate the generalization behaviors of CNN from a data perspective.”
- Multi-attention Recurrent Network for Human Communication Comprehension (2018, methodology): “SAL-CNN (Selective Additive Learning CNN) (Wang et al. 2016) is a model that attempts to prevent identitydependent information from being learned by using Gaussian corruption introduced to the neuron outputs.”
- Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment (2018, methodology): “To further make use of the combined features for classification, we applied a CNN structure with one convolutional layer and one max-pooling layer to extract the final representation from shared word-level vectors (Poria et al., 2016; Wang et al., 2016).”
- Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors (2018, methodology): “While these methods are able to out-perform unimodal models (Zadeh et al. 2016) and learn robust representations (Wang et al. 2016), they have limited capabilities in learning modality-specific interactions and tend to overfit (Xu, Tao, and Xu 2013).”
- Tensor Fusion Network for Multimodal Sentiment Analysis (2017, background): “The newest multimodal sentiment analysis approaches have used deep neural networks, including convolutional neural networks (CNNs) with multiple-kernel learning (Poria et al., 2015), SAL-CNN (Wang et al., 2016) which learns generalizable features across speakers, and support vector machines (SVMs) w”
- SMIL: Multimodal Learning with Severely Missing Modality (2021, methodology): “Early fusion is a common method which fuses different modalities by feature concatenation, and it has been widely adopted in previous studies (Wang et al. 2017; Poria et al. 2016).”
- Efficient Low-rank Multimodal Fusion With Modality-Specific Factors (2018, methodology): “Several works that use this method of fusion (Poria et al., 2016) , (Wang et al., 2016) use input-level feature concatenation and use the concatenated features as input, sometimes even removing the temporal dependency present in the modalities (Morency et al., 2011).”
- Memory Fusion Network for Multi-view Sequential Learning (2018, background): “Wang2016 (∩): Selective Additive Learning Convolutional Neural Network (Wang et al. 2016) is a multimodal sentiment analysis model that attempts to prevent identity-dependent information from being learned so as to improve generalization based only on accurate indicators of sentiment.”
- Multimodal Sentiment Analysis: A Survey of Methods, Trends, and Challenges (2023, methodology): “[167] proposed a Select-Additive Learning (SAL) procedure to improve the generalizability of trained neural networks for multimodal sentiment analysis.”
- Multimodal Language Analysis with Recurrent Multistage Fusion (2018, methodology): “For descriptions of the remaining baselines, we refer the reader to EF-HCRF (Quattoni et al., 2007), EF/MV-LDHCRF (Morency et al., 2007), MV-HCRF (Song et al., 2012), EF/MVHSSHCRF (Song et al., 2013), MV-LSTM (Rajagopalan et al., 2016), DF (Nojavanasghari et al., 2016), SAL-CNN (Wang et al., 2016), ”
- Conversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos (2018, background): “Recently, this combination of modalities has provided the best performance in affect recognition systems (Poria et al., 2017b; Wang et al., 2017; Tzirakis et al., 2017), thus motivating the use of a multimodal approach.”