Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors

By Yansen Wang, Ying Shen, Zhun Liu, Percy Liang, Amir Zadeh, Louis-philippe Morency. 2018

490 citations recorded by the source corpus.

Humans convey their intentions through the usage of both verbal and nonverbal behaviors during face-to-face communication. Speaker intentions often vary dynamically depending on different nonverbal contexts, such as vocal patterns and facial expressions. As a result, when modeling human language, it is essential to not only consider the literal meaning of the words but also the nonverbal contexts in which these words appear. To better model human language, we first model expressive nonverbal representations by analyzing the fine-grained visual and acoustic patterns that occur during word segments. In addition, we seek to capture the dynamic nature of nonverbal intents by shifting word representations based on the accompanying nonverbal behaviors. To this end, we propose the Recurrent Attended Variation Embedding Network (RAVEN) that models the fine-grained structure of nonverbal subword sequences and dynamically shifts word representations based on nonverbal cues. Our proposed model achieves competitive performance on two publicly available datasets for multimodal sentiment analysis and emotion recognition. We also visualize the shifted word representations in different nonverbal contexts and summarize common patterns regarding multimodal variations of word representations.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Experimental and Cognitive Psychology

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Multimodal Transformer for Unaligned Multimodal Language Sequences (2019, methodology): “…Early Fusion LSTM (EF-LSTM) and Late Fusion LSTM (LF-LSTM) as baseline models, as well as Recurrent Attended Variation Embedding Network (RAVEN) (Wang et al., 2019) and Multimodal Cyclic Translation Network (MCTN) (Pham et al., 2019), that achieved SOTA results on various word-aligned human 1 2…”
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis (2020, unknown): “• Models which utilize attention and transformer modules to improve token representations using non-verbal signals:RAVEN [56], MulT [52].”
Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions (2022, background): “[49] RAVEN 2019 Word-level Fusion RAVEN CMU-MOSI Recurrent Attended Variation Embedding Network (RAVEN)”
Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis (2021, methodology): “For representation learning methods, Wang et al. (2019) constructed a recurrent attended variation embedding network to generate multimodal shifting.”
Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis (2019, background): “Other approaches (Zadeh et al. 2018a; Liang et al. 2018; Zadeh et al. 2018c; Wang et al. 2018) study multi-modal interactions and intra-actions by using either graph or temporal memory networks with a sequential neural network LSTM (Gers, Schmidhu-ber, and Cummins 1999).”
Integrating Multimodal Information in Large Pretrained Transformers (2020, background): “A particularly appealing implementation of such displacement is studied in RAVEN (Wang et al., 2018), where displacements are calculated using cross-modal self-attention to highlight relevant non-verbal information.”
Factorized Multimodal Transformer for Multimodal Sequential Learning (2019, result): “Multiple FMS can be used inside a MTL to efﬁciently extract diverse multimodal Model \ Metric BA F1 MAE Corr MV-LSTM (Rajagopalan et al., 2016) 73.9/– 74.0/– 1.019 0 .601 TFN (Zadeh et al., 2017) 73.9/– 73.4/– 1.040 0 .633 MARN (Zadeh et al., 2018b) 77.1/– 77.0/– 0.968 0.625 MFN (Zadeh et al., 2018a”