What Makes Training Multi-Modal Classification Networks Hard?

By Weiyao Wang, Du Tran, Matt Feiszli. 2019

665 citations recorded by the source corpus.

Consider end-to-end training of a multi-modal vs. a uni-modal network on a task with multiple input modalities: the multi-modal network receives more information, so it should match or outperform its uni-modal counterpart. In our experiments, however, we observe the opposite: the best uni-modal network can outperform the multi-modal network. This observation is consistent across different combinations of modalities and on different tasks and benchmarks for video classifications. This paper identifies two main causes for this performance drop: first, multi-modal networks are often prone to overfitting due to increased capacity. Second, different modalities overfit and generalize at different rates, so training them jointly with a single optimization strategy is sub-optimal. We address these two problems with a technique we call Gradient-Blending, which computes an optimal blending of modalities based on their overfitting behaviors. We demonstrate that Gradient Blending outperforms widely-used baselines for avoiding overfitting and achieves state-of-the-art accuracy on various tasks including human action recognition, ego-centric action recognition, and acoustic event detection.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Perceiver: General Perception with Iterative Attention (2021, methodology): “Audio+video fusion leads to solid improvements over single modalities (and outperforms specialized fusion optimization approaches (Wang et al., 2020c)) but is still lower than the state-of-the-art approach that uses separate models with late fusion (Fayek & Kumar, 2020).”
Supervised Multimodal Bitransformers for Classifying Images and Text (2019, methodology): “In NLP it is common to ﬁne-tune BERT in its entirety, and not to transfer the encoder while keeping its parameters ﬁxed, as used to be the case in e.g. Training multimodal models is not at all trivial, especially when it comes to the optimization strategy (Wang, Tran, and Feiszli 2019).”
Attention Bottlenecks for Multimodal Fusion (2021, background): “…designing a uniﬁed model for modality fusion is challenging due to a number of factors: (i) variations in learning dynamics between modalities [63], (ii) different noise topologies, with some modality streams containing more information for the task at hand than others, as well as (iii)…”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “However, as pointed out by [417], many existing multimodality methods are not as effective as expected owing to”
Self-Supervised Learning by Cross-Modal Audio-Video Clustering (2019, background): “Also, recent work [74] has shown that models trained on different modalities learn and generalize at different speeds, and that training them jointly (as done in MDC which uses two-modality heads) is sub-optimal.”
Balanced Multimodal Learning via On-the-fly Gradient Modulation (2022, methodology): “Recently, several methods have emerged trying to solve these problems [8, 35, 39].”
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (2021, background): “Model architectures proposed for these tasks include multi-stream networks [46, 38, 35, 68], recurrent networks [22, 21, 65], 3D conv nets [53, 44] and spatially grounded topological graph models [50].”
Harnessing multimodal data integration to advance precision oncology (2021, methodology): “One mechanism to address this is to incorporate the estimated generalization error in the training objective, using techniques such as gradient blending, a technique to weight each unimodal contribution to the overall loss on the basis of its estimated generalization error 62 .”
Multimodal deep learning for biomedical data fusion: a review (2022, background): “Overfitting to the training data, and therefore poor generalizability, is a major challenge for multimodal models [103].”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, background): “Recent works have also found that multi-modal learning can lead to more robust representations as they can partly account for the different learning speeds of the different modalities [2, 64, 61].”
What Makes Multimodal Learning Better than Single (Provably) (2021, methodology): “It has been discovered that the use of multi-modal data in practice will degrade the performance of the model in some cases[43, 19, 38, 15].”
Preserving integrity in online social networks (2020, background): “Gradient blending [127] has recently been introduced address this issue.”
One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting (2020, background): “Several recent studies [31], [53], [61] have shown that the interactions between networks can degrade the performance of a whole network, and that catastrophic forgetting with one-shot NAS can lower the performance of previous architectures after training a new”
Trusted Multi-View Classification With Dynamic Evidential Fusion (2022, methodology): “used in various applications [47], [48], [49], [50].”