Multi-attention Recurrent Network for Human Communication Comprehension

By Amir Zadeh, Percy Liang, Soujanya Poria, Prateek Vij, Erik Cambria, Louis-philippe Morency. AAAI Conference on Artificial Intelligence, 2018

575 citations recorded by the source corpus.

Human face-to-face communication is a complex multimodal signal. We use words (language modality), gestures (vision modality) and changes in tone (acoustic modality) to convey our intentions. Humans easily process and understand face-to-face communication, however, comprehending this form of communication remains a significant challenge for Artificial Intelligence (AI). AI must understand each modality and the interactions between them that shape the communication. In this paper, we present a novel neural architecture for understanding human communication called the Multi-attention Recurrent Network (MARN). The main strength of our model comes from discovering interactions between modalities through time using a neural component called the Multi-attention Block (MAB) and storing them in the hybrid memory of a recurrent component called the Long-short Term Hybrid Memory (LSTHM). We perform extensive comparisons on six publicly available datasets for multimodal sentiment analysis, speaker trait recognition and emotion recognition. MARN shows state-of-the-art results performance in all the datasets.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, Signal Processing

View Paper: https://www.semanticscholar.org/paper/90f3bd3141026e3a15358149e7de42a3c7ed7f31

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Experimental and Cognitive Psychology

In Signal Processing

Citations in the corpus

COSMIC: COmmonSense knowledge for eMotion Identification in Conversations (2020, background): “2472 important works include (Hazarika et al., 2018a,b; Zadeh et al., 2018b; Chen et al., 2017; Zadeh et al., 2018a).”
Factorized Multimodal Transformer for Multimodal Sequential Learning (2019, methodology): “We use two approaches for calculating BA and F1 based on negative vs. non-negative sentiment (Zadeh et al., 2018b) on the left side of /, and negative vs. positive (Tsai et al., 2019) on the right side.”
MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations (2018, background): “The other publicly available multimodal emotion and sentiment recognition datasets are MOSEI (Zadeh et al., 2018b), MOSI (Zadeh et al., 2016b), and MOUD (P´erez-Rosas et al., 2013), however none of these datasets is conversational.”
Efficient Low-rank Multimodal Fusion With Modality-Specific Factors (2018, methodology): “Multi-attention Recurrent Network The Multi-attention Recurrent Network (MARN) (Zadeh et al., 2018b) explicitly models interactions between modalities through time using a neural component called the Multi-attention Block (MAB) and storing them in the hybrid memory called the Long-short Term Hybrid ”
Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis (2019, methodology): “Among multi-modal language sentiment or emotion experiments involving unimodal features (Zadeh et al. 2016; Zadeh et al. 2018c; Tsai et al. 2018; Tsai et al. 2019), it is commonly observed that text based features perform better than visual or auditory modes.”
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities (2018, unknown): “Most work follows this framework during both training and testing (Liang et al. 2018; Liu et al. 2018; Tsai et al. 2018; Zadeh et al. 2018).”
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis (2020, background): “• Networks which perform temporal modeling and fusion of utterances:MFN [59],MARN [61],MV-LSTM [46], RMFN [24].”
Multimodal deep learning for biomedical data fusion: a review (2022, methodology): “[73] used an attention mechanism for multiomics data [99] for interpreting predictions of gene expression.”
Multimodal Language Analysis with Recurrent Multistage Fusion (2018, methodology): “We compare to the following models for multimodal machine learning: MFN (Zadeh et al., 2018a) synchronizes multimodal sequences using a multi-view gated memory.”
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph (2018, methodology): “We replace the Delta-memory Attention Network with DFG and refer to the modiﬁed model as Graph Memory Fusion Network (Graph-MFN).”
DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation (2019, background): “Some of the important works include (Poria et al., 2017; Chen et al., 2017; Zadeh et al., 2018a,b; Hazarika et al., 2018a,b), where mainly deep learning-based techniques have been employed for emotion (and sentiment) recognition in conversation, in only textual and multimodal settings.”
Learning Factorized Multimodal Representations (2018, methodology): “For a detailed description of the baselines, we point the reader to MFN (Zadeh et al., 2018a), MARN (Zadeh et al., 2018b), TFN (Zadeh et al., 2017), BC-LSTM (Poria et al., 2017), MVLSTM (Rajagopalan et al., 2016), EF-LSTM (Hochreiter & Schmidhuber, 1997; Graves et al., 2013; Schuster & Paliwal,…”
Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research (2020, methodology): “[236], [237], [239] have been recently proposed, in our experience, a simple concatenation-based fusion method performs at par with most of these methods.”
DialogueRNN: An Attentive RNN for Emotion Detection in Conversations (2018, methodology): “MFN (Zadeh et al. 2018a): Speciﬁc to multimodal scenario, this model utilizes multi-view learning by modeling view-speciﬁc and cross-view interactions.”
ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection (2018, methodology): “In emotional analysis, Zadeh et al. (2018a) propose a memory-based sequential learning for multi-view signals.”