Efficient Low-rank Multimodal Fusion With Modality-Specific Factors

By Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Percy Liang, Amir Zadeh, Louis-philippe Morency. Annual Meeting of the Association for Computational Linguistics, 2018

995 citations recorded by the source corpus.

Multimodal research is an emerging field of artificial intelligence, and one of the main research problems in this field is multimodal fusion. The fusion of multimodal data is the process of integrating multiple unimodal representations into one compact multimodal representation. Previous research in this field has exploited the expressiveness of tensors for multimodal representation. However, these methods often suffer from exponential increase in dimensions and in computational complexity introduced by transformation of input into tensor. In this paper, we propose the Low-rank Multimodal Fusion method, which performs multimodal fusion using low-rank tensors to improve efficiency. We evaluate our model on three different tasks: multimodal sentiment analysis, speaker trait analysis, and emotion recognition. Our model achieves competitive results on all these tasks while drastically reducing computational complexity. Additional experiments also show that our model can perform robustly for a wide range of low-rank settings, and is indeed much more efficient in both training and inference compared to other methods that utilize tensor representations.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/85653b72209fbf6cb0b9d4f5da2be4d35678ec73

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis (2019, methodology): “An alternative is to use the outer product (Liu et al. 2018; Zadeh et al. 2017) which can represent the interaction between pairs of features, resulting in 2D or 3D arrays that can be processed using advanced methods such as convolutional neural networks (CNNs) (Lawrence et al. 1997).”
Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis (2021, methodology): “We can find that the Self-MM gets better results than TFN and LMF and achieve comparable performance with Human-MM.”
Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence (2019, methodology): “creatively used by research communities centered around tasks such as multimodal language analysis [58], sentiment analysis [46, 31], emotion recognition [29], personality traits recognition [56], image captioning [5, 4, 34], multimedia description [50, 51, 62], and video comprehension [14, 32, 19, ”
Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors (2018, methodology): “2018), and more expressive fusion methods (Liu et al. 2018).”
Multi-modal Knowledge Graphs for Recommender Systems (2020, background): “For example, it is natural to explore more ways of multi-modal fusion under the framework of multi-modal knowledge graph, such as Tensor Fusion Network (TFN) [34] or Low-rank Multimodal Fusion (LMF) [14].”
What Makes Training Multi-Modal Classification Networks Hard? (2019, methodology): “On a different fusion architecture with Low-Rank Multi-Modal Fusion (LMF) [35], GradientBlending gives 4.”
i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data (2023, background): “For example, Wang et al. (2022c); Lu et al. (2022); Tang et al. (2022) unite vision and vision-language tasks, such as image classification, object detection, semantic segmentation, visual QA, document understanding, image generation, etc. Huang et al. (2023) recently proposed the Kosmos-1 model to…”
Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis (2021, methodology): “The latter additionally rectifies the spatial orientation of unimodal or multimodal representations by matrix decomposition (Liu et al., 2018) or Euclidean measure optimization (Sun et al.”
Deep Multimodal Data Fusion (2024, background): “Furthermore, there are variations of this work, e.g., [108].”
Harnessing multimodal data integration to advance precision oncology (2021, background): “Advances in multimodal ML methods have been impressive in other fields, such as sentiment analysis 86,149–154 , with large benchmark datasets, but the largest multimodal oncological dataset, the TCGA dataset, contains limited data modalities and only a few hundred patients per cancer type.”
Multimodal Language Analysis with Recurrent Multistage Fusion (2018, background): “Modeling these interactions lie at the heart of human multimodal language analysis and has recently become a centric research direction in multimodal natural language processing (Liu et al., 2018; Pham et al., 2018; Chen et al., 2017), multimodal speech recognition (Sun et al.”
Learning Factorized Multimodal Representations (2018, background): “For instance, recent works learn visual representations that are maximally dependent with linguistic attributes for improving one-shot image recognition (Tsai & Salakhutdinov, 2017) or introduce tensor product mechanisms to model interactions between the language, visual and acoustic modalities (Liu”
A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2021, methodology): “In the case of a late fusion, such techniques as majority vote [38] and low-rank multimodal fusion [39] may be used to aggregate the final prediction scores of several classifiers.”
SMIL: Multimodal Learning with Severely Missing Modality (2021, methodology): “We follow Liu et al. (2018) to get features for the image and text modality.”
Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos (2020, methodology): “We note that modality interaction is common in VQA methods, but they usually pool the multimodal feature sequences into a single vector using bilinear or multi-linear pooling [16,30,31,35].”
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities (2018, neutral): “Most work follows this framework during both training and testing (Liang et al. 2018; Liu et al. 2018; Tsai et al. 2018; Zadeh et al. 2018).”
Factorized Multimodal Transformer for Multimodal Sequential Learning (2019, methodology): “, 2017) and its approximate variants (Liang et al.; Liu et al., 2018), as well as Compact Bilinear Pooling (Gao et al.”
Exploring Deep Multimodal Fusion of Text and Photo for Hate Speech Classification (2019, methodology): “For the task of sentiment analysis, Zadeh et al. (2017); Ghosal et al. (2018); Bagher Zadeh et al. (2018); Liu et al. (2018) propose several models, namely contextual intermodal attention, dynamic fusion graph, and lowrank multimodal fusion, for integrating visual, audio, and text signals on the…”