Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment

By Yue Gu, Kangning Yang, Shiyu Fu, Shuhong Chen, Xinyu Li, Ivan Marsic. 2018

144 citations recorded by the source corpus.

Multimodal affective computing, learning to recognize and interpret human affect and subjective information from multiple data sources, is still a challenge because: (i) it is hard to extract informative features to represent human affects from heterogeneous inputs; (ii) current fusion strategies only fuse different modalities at abstract levels, ignoring time-dependent interactions between modalities. Addressing such issues, we introduce a hierarchical multimodal architecture with attention and word-level fusion to classify utterance-level sentiment and emotion from text and audio data. Our introduced model outperforms state-of-the-art approaches on published datasets, and we demonstrate that our model is able to visualize and interpret synchronized attention over modalities.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Experimental and Cognitive Psychology

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Multimodal Transformer for Unaligned Multimodal Language Sequences (2019, methodology): “In comparison, one common way of tackling unaligned multimodal sequence is by forced word-aligning before training (Poria et al., 2017; Zadeh et al., 2018a,b; Tsai et al., 2019; Pham et al., 2019; Gu et al., 2018): manually preprocess the visual and acoustic features by aligning them to the resoluti”
Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model (2019, methodology): “Gu et al. (2018b) align text and audio at word level and apply several attention mechanisms.”
Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors (2018, methodology): “Following prior practice (Liu et al. 2018; Liang et al. 2018; Gu et al. 2018), we adopted the same feature extraction scheme for language, visual and acoustic modalities.”