Multimodal Language Analysis with Recurrent Multistage Fusion

By Percy Liang, Liu Ziyin, Amir Zadeh, Louis-philippe Morency. 2018

222 citations recorded by the source corpus.

Computational modeling of human multimodal language is an emerging research area in natural language processing spanning the language, visual and acoustic modalities. Comprehending multimodal language requires modeling not only the interactions within each modality (intra-modal interactions) but more importantly the interactions between modalities (cross-modal interactions). In this paper, we propose the Recurrent Multistage Fusion Network (RMFN) which decomposes the fusion problem into multiple stages, each of them focused on a subset of multimodal signals for specialized, effective fusion. Cross-modal interactions are modeled using this multistage fusion approach which builds upon intermediate representations of previous stages. Temporal and intra-modal interactions are modeled by integrating our proposed fusion approach with a system of recurrent neural networks. The RMFN displays state-of-the-art performance in modeling human multimodal language across three public datasets relating to multimodal sentiment analysis, emotion recognition, and speaker traits recognition. We provide visualizations to show that each stage of fusion focuses on a different subset of multimodal signals, learning increasingly discriminative multimodal representations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Multimodal Transformer for Unaligned Multimodal Language Sequences (2019, methodology): “Classical crossmodal alignment, on the other hand, can be expressed as a special (step di-Table (Liang et al., 2018) 38.3 78.4 78.0 0.922 0.681 MFM (Tsai et al., 2019) 36.2 78.1 78.1 0.951 0.662 RAVEN (Wang et al., 2019) 33.2 78.0 76.6 0.915 0.691 MCTN (Pham et al., 2019) 35 (Zadeh et al., 2018b)…”
Towards Debiasing Sentence Representations (2020, neutral): “These datasets have been the subject of recent research in language understanding (Merity et al., 2017b; Liu et al., 2019; Wang et al., 2019) and multimodal human language (Liang et al., 2018, 2019).”
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis (2020, methodology): “ver representations of utterances, another line of work takes a fine-grained view to perform fusion at the word level.Approachesincludemultimodal-awarewordembeddings[56], recurrent multi-stage fusion [24], graph-based fusion [30, 60], recurrent networks (RNNs), attention-models, memory mechanisms, a”
Factorized Multimodal Transformer for Multimodal Sequential Learning (2019, methodology): “We perform experiments for discrete emotions (Ekman, 1992) (Zadeh et al., 2017) 24.1 31.0 31.5 34.5 24.6 25.6 27.6 29.1 MARN (Zadeh et al., 2018b) 29.1 33.0 - - 31.5 - - - MFN (Zadeh et al., 2018a) 34.5 35.5 37.4 41.9 34.5 36.9 36.0 37.9 RMFN (Liang et al., 2018) 37.4 38.4 37.4 - 37.4 38.9 38.9 - Mu”
Integrating Multimodal Information in Large Pretrained Transformers (2020, background): “LSTMs in TFN, MARN, MFN, RMFN, LFN use latent size of { 16 , 32 , 64 , 128 } .”
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities (2018, unknown): “(Liang et al. 2018) presented a multistage approach to learn hierarchical multimodal representations.”
Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions (2022, methodology): “[48] RMFN 2018 Word-level Fusion LSTHM CMU-MOSI Gated Modality mixing Network Multimodal Shifting 85.”
Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence (2019, methodology): “…used by research communities centered around tasks such as multimodal language analysis [58], sentiment analysis [46, 31], emotion recognition [29], personality traits recognition [56], image captioning [5, 4, 34], multimedia description [50, 51, 62], and video comprehension [14, 32, 19, 60].”
Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis (2019, background): “Uni-modal features can be extracted using a variety of simple extraction schemes as well as by learning features using more complex neural network based models such as a sequential LSTM.”
Neural Networks Fail to Learn Periodic Functions and How to Fix It (2020, background): “This kind of data naturally appears in many audio and textual tasks, such as machine translation [4], audio generation [14], or multimodal learning [23].”