Memory Fusion Network for Multi-view Sequential Learning

By Amir Zadeh, Percy Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, Louis-philippe Morency. 2018

940 citations recorded by the source corpus.

Multi-view sequential learning is a fundamental problem in machine learning dealing with multi-view sequences. In a multi-view sequence, there exists two forms of interactions between different views: view-specific interactions and cross-view interactions. In this paper, we present a new neural architecture for multi-view sequential learning called the Memory Fusion Network (MFN) that explicitly accounts for both interactions in a neural architecture and continuously models them through time. The first component of the MFN is called the System of LSTMs, where view-specific interactions are learned in isolation through assigning an LSTM function to each view. The cross-view interactions are then identified using a special attention mechanism called the Delta-memory Attention Network (DMAN) and summarized through time with a Multi-view Gated Memory. Through extensive experimentation, MFN is compared to various proposed approaches for multi-view sequential learning on multiple publicly available benchmark datasets. MFN outperforms all the multi-view approaches. Furthermore, MFN outperforms all current state-of-the-art models, setting new state-of-the-art results for all three multi-view datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Multimodal Transformer for Unaligned Multimodal Language Sequences (2019, methodology): “Both tasks are frequently used to benchmark models’ ability to fuse multimodal (sentiment) information (Poria et al., 2017; Zadeh et al., 2018a; Liang et al., 2018; Tsai et al., 2019; Pham et al., 2019; Wang et al., 2019).”
DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation (2019, background): “Some of the important works include (Poria et al., 2017; Chen et al., 2017; Zadeh et al., 2018a,b; Hazarika et al., 2018a,b), where mainly deep learning-based techniques have been employed for emotion (and sentiment) recognition in conversation, in only textual and multimodal settings.”
Multimodal Language Analysis with Recurrent Multistage Fusion (2018, methodology): “, 2018), external memory mechanisms to synchronize multimodal data (Zadeh et al., 2018a), or lowrank tensors to approximate expensive tensor products (Liu et al.”
Harnessing multimodal data integration to advance precision oncology (2021, background): “Advances in multimodal ML methods have been impressive in other fields, such as sentiment analysis 86,149–154 , with large benchmark datasets, but the largest multimodal oncological dataset, the TCGA dataset, contains limited data modalities and only a few hundred patients per cancer type.”
DialogueRNN: An Attentive RNN for Emotion Detection in Conversations (2018, methodology): “MFN (Zadeh et al. 2018a): Speciﬁc to multimodal scenario, this model utilizes multi-view learning by modeling view-speciﬁc and cross-view interactions.”
Factorized Multimodal Transformer for Multimodal Sequential Learning (2019, methodology): “We perform experiments for discrete emotions (Ekman, 1992) (Zadeh et al., 2017) 24.1 31.0 31.5 34.5 24.6 25.6 27.6 29.1 MARN (Zadeh et al., 2018b) 29.1 33.0 - - 31.5 - - - MFN (Zadeh et al., 2018a) 34.5 35.5 37.4 41.9 34.5 36.9 36.0 37.9 RMFN (Liang et al., 2018) 37.4 38.4 37.4 - 37.4 38.9 38.9 - Mu”
Learning Factorized Multimodal Representations (2018, methodology): “For a detailed description of the baselines, we point the reader to MFN (Zadeh et al., 2018a), MARN (Zadeh et al., 2018b), TFN (Zadeh et al., 2017), BC-LSTM (Poria et al., 2017), MVLSTM (Rajagopalan et al., 2016), EF-LSTM (Hochreiter & Schmidhuber, 1997; Graves et al., 2013; Schuster & Paliwal,…”
Audio-Visual Fusion for Sentiment Classiﬁcation using Cross-Modal Autoencoder (2018, methodology): “In this paper, we consider the task of multi-modal sentiment classification from videos which have both visual and audio modalities1 [4, 5, 6, 7].”
Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions (2022, background): “[46] MFN 2018 Word-level Fusion MFN MOUD MFN shows a consistent trend for both classification and regression 80.”
Integrating Multimodal Information in Large Pretrained Transformers (2020, methodology): “MFN (Memory Fusion Network) has three separate LSTMs to model each modality separately and a multi-view gated memory to synchronize among them (Zadeh et al., 2018a).”
MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation (2021, methodology): “However, MFN neglects to model speaker information, which is signiﬁcant to ERC as well.”
Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis (2019, methodology): “Among multi-modal language sentiment or emotion experiments involving unimodal features (Zadeh et al. 2016; Zadeh et al. 2018c; Tsai et al. 2018; Tsai et al. 2019), it is commonly observed that text based features perform better than visual or auditory modes.”
Emotions Don't Lie: An Audio-Visual Deepfake Detection Method using Affective Cues (2020, background): “F2 and S2 are based on the Memory Fusion Network (MFN) [62], which is reported to have SOTA performance on both emotion recognition from multiple views or modalities like face and speech.”
Sentiment Analysis using Imperfect Views from Spoken Language and Acoustic Modalities (2018, background): “…existing and popularly used representations of the acoustic view have generally under-performed compared to the language view (Poria et al., 2017; Zadeh et al., 2018; P´erez-Rosas et al., 2013), indicating that the acoustic view or its representations, by themselves, may not be discriminative…”
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis (2020, background): “• Networks which perform temporal modeling and fusion of utterances:MFN [59],MARN [61],MV-LSTM [46], RMFN [24].”
ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection (2018, methodology): “In emotional analysis, Zadeh et al. (2018a) propose a memory-based sequential learning for multi-view signals.”