Multimodal Transformer for Unaligned Multimodal Language Sequences

By Yao-Hung Hubert Tsai, Shaojie Bai, Percy Liang, J. Kolter, Louis-philippe Morency, Ruslan Salakhutdinov. 2019

2,255 citations recorded by the source corpus.

Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. However, two major challenges in modeling such multimodal human language time-series data exist: 1) inherent data non-alignment due to variable sampling rates for the sequences from each modality; and 2) long-range dependencies between elements across modalities. In this paper, we introduce the Multimodal Transformer (MulT) to generically address the above issues in an end-to-end manner without explicitly aligning the data. At the heart of our model is the directional pairwise crossmodal attention, which attends to interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another. Comprehensive experiments on both aligned and non-aligned multimodal time-series show that our model outperforms state-of-the-art methods by a large margin. In addition, empirical analysis suggests that correlated crossmodal signals are able to be captured by the proposed crossmodal attention mechanism in MulT.

Subfields: Artificial Intelligence, Multimodal Learning, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Multimodal Learning

In Model Architectures

In Signal Processing

Citations in the corpus

Experience Grounds Language (2020, background): “, 2018)) or further broadened to include audio (Tsai et al., 2019).”
Self-Supervised MultiModal Versatile Networks (2020, background): “Using audio, vision and language to learn representations has also been explored in past work [11, 31, 38, 50, 83].”
Scaling Laws for Autoregressive Generative Modeling (2020, methodology): “Transformer-based video models were trained in [WTU19] and multimodal models in [TBL + 19].”
Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text (2020, background): “BERT+ cross-modal self-attention We also consider a cross-modal self-attention layer of the type suggested by e.g. Tsai et al. [2019], Lu et al. [2019], which in our application provides an explicit pathway for the agent to bind visual experience to speciﬁc (contextual) word representations.”
Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis (2021, methodology): “…h t ∈ R d t ( d t =768) to lower ones h v ∈ R d v and h a ∈ R d a ( d v , d a ¡ 50) converges faster with higher accuracy; 2) many previous works (Tsai et al., 2019a; Hazarika et al., 2020) pointed out that from empirical study the text modality is predominate, which can integrate more…”
Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis (2019, background): “A multimodal transformer is proposed by (Tsai et al. 2019) that uses attention based cross-modal transformers to learn interactions between modalities.”
Factorized Multimodal Transformer for Multimodal Sequential Learning (2019, methodology): “We use two approaches for calculating BA and F1 based on negative vs. non-negative sentiment (Zadeh et al., 2018b) on the left side of /, and negative vs. positive (Tsai et al., 2019) on the right side.”
MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation (2021, methodology): “To verify the effectiveness of MMGCN in multi-modal fusion, we compare it with other multimodal fusion methods, including early fusion, late fusion, fusion through gated attention and other representative fusion methods such as MFN(Zadeh et al., 2018) and MulT(Tsai et al., 2019).”
Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021, background): “Previous works have shown that these different modalities are complimentary for emotion expression, and proposed many effective multimodal fusion methods to improve the emotion recognition performance (Baltruˇsaitis et al., 2018; Tsai et al., 2019; Zhao et al., 2018).”
CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality (2020, methodology): “The Multimodal Transformer (MULT) (Tsai et al., 2019) using the directional pairwise crossmodal attention to realize the interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another.”
A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation (2024, methodology): “More recently, the transformer architecture has been considered for multimodal fusion in semantic segmentation for its outstanding capability in extracting contextual information [20], [26], [32], [54] and fusing different modalities [19], [38], [55], [56], [57], [58], [59].”
Multimodal and Multilingual Embeddings for Large-Scale Speech Mining (2021, background): “At the sentence level, text-audio sentiment analysis often intrinsically introduces a fixed-size crossmodal representation before classifying the input, as in (Yang et al., 2020; Tsai et al., 2019).”
Multimodal Learning With Transformers: A Survey (2022, background): “This kind of hierarchically cross-modal interaction is also widely studied [137], [189], and alleviates the drawback of cross-attention.”
Trusted Multi-View Classification With Dynamic Evidential Fusion (2022, background): “For multi-view learning, traditional methods usually assume an equal value for different views [10], [11], [12], [13] or learn a fixed weight factor for each [14], [15], [16], [17], [18].”
Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel (2019, methodology): “…allows us to explore new family of attention while at the same time offering a framework to categorize previous attention variants (Vaswani et al., 2017; Shaw et al., 2018; Huang et al., 2018b; Dai et al., 2019; Child et al., 2019; Lee et al., 2018; Wang et al., 2018; Tsai et al., 2019a).”
Visual Grounding with Transformers (2021, background): “[37] introduce a cross-modal attention mechanism for fusing information from different modalities.”