What Makes Multimodal Learning Better than Single (Provably)
By Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao. Neural Information Processing Systems, 2021
383 citations recorded by the source corpus.
The world provides us with data of multiple modalities. Intuitively, models fusing data from different modalities outperform their uni-modal counterparts, since more information is aggregated. Recently, joining the success of deep learning, there is an influential line of work on deep multi-modal learning, which has remarkable empirical results on various applications. However, theoretical justifications in this field are notably lacking. Can multi-modal learning provably perform better than uni-modal? In this paper, we answer this question under a most popular multi-modal fusion framework, which firstly encodes features from different modalities into a common latent space and seamlessly maps the latent representations into the task space. We prove that learning with multiple modalities achieves a smaller population risk than only using its subset of modalities. The main intuition is that the former has a more accurate estimate of the latent space representation. To the best of our knowledge, this is the first theoretical treatment to capture important qualitative phenomena observed in real multi-modal applications from the generalization perspective. Combining with experiment results, we show that multi-modal learning does possess an appealing formal guarantee.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/0a60746d598bae9f0ce6f25f68f4fcf668a9f13b
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
Citations in the corpus
- M5Product: A Multi-modal Pretraining Benchmark for E-commercial Product Downstream Tasks (2021, background): “Recent progress in multi-modal pre-training, from both theoretical [1] and practical [2, 3] perspectives, proves that large scale samples with diverse modalities can effectively enhance the discrimination of generated features and thus improve the performance in the vision-language tasks.”
- Agentic AI: Autonomous Intelligence for Complex Goals—A Comprehensive Survey (2025, neutral): “Furthermore, investigation of multi-modal learning [94], [95], which fuses information from multiple sources and types to better understand different situations, may assist in improving the versatility of Agentic AI systems in the operative context.”
- Self-Supervised Remote Sensing Feature Learning: Learning Paradigms, Challenges, and Future Works (2022, background): “[216] theoretically and experimentally demonstrated that the more modal data used, the more likely the deep learning model to gain high-quality complementary features and achieve better performance in classification tasks.”