Integrating Multimodal Information in Large Pretrained Transformers

By Wasifur Rahman, Md. Kamrul Hasan, Sangwu Lee, Amir Zadeh, Chengfeng Mao, Louis-philippe Morency, Ehsan Hoque. 2020

823 citations recorded by the source corpus.

Recent Transformer-based contextual word representations, including BERT and XLNet, have shown state-of-the-art performance in multiple disciplines within NLP. Fine-tuning the trained contextual models on task-specific datasets has been the key to achieving superior performance downstream. While fine-tuning these pre-trained models is straightforward for lexical applications (applications with only language modality), it is not trivial for multimodal language (a growing area in NLP focused on modeling face-to-face communication). More specifically, this is due to the fact that pre-trained models don’t have the necessary components to accept two extra modalities of vision and acoustic. In this paper, we proposed an attachment to BERT and XLNet called Multimodal Adaptation Gate (MAG). MAG allows BERT and XLNet to accept multimodal nonverbal data during fine-tuning. It does so by generating a shift to internal representation of BERT and XLNet; a shift that is conditioned on the visual and acoustic modalities. In our experiments, we study the commonly used CMU-MOSI and CMU-MOSEI datasets for multimodal sentiment analysis. Fine-tuning MAG-BERT and MAG-XLNet significantly boosts the sentiment analysis performance over previous baselines as well as language-only fine-tuning of BERT and XLNet. On the CMU-MOSI dataset, MAG-XLNet achieves human-level multimodal sentiment analysis performance for the first time in the NLP community.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis (2021, methodology): “The Multimodal Adaptation Gate for Bert (MAG-BERT) (Rahman et al. 2020) is an improvement over RAVEN on aligned data with applying multimodal adaptation gate at different layers of the BERT backbone.”
Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions (2022, background): “Finetuning MAGBERT and MAG-XLNet, as well as language-only finetuning of BERT and XLNet, greatly improves sentiment analysis performance over earlier baselines [93].”
Span-based Localizing Network for Natural Language Video Localization (2020, methodology): “Meanwhile, similar pre-trained models (Sun et al., 2019a,b; Yu and Jiang, 2019; Rahman et al., 2019; Nguyen and Okatani, 2019; Lu et al., 2019b; Tan and Bansal, 2019) are being proposed to learn joint distributions over multimodality sequence of visual and linguistic inputs.”
ChatGPT: Jack of all trades, master of none (2023, neutral): “As more powerful GPUs and TPUs were developed [9], it became possible to create models with more and more parameters, resulting in models that began to achieve human performance for an increasing number of tasks [10, 11, 12].”