Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning

By Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, James Y. Zou. Neural Information Processing Systems, 2022

774 citations recorded by the source corpus.

We present modality gap, an intriguing geometric phenomenon of the representation space of multi-modal models. Specifically, we show that different data modalities (e.g. images and text) are embedded at arm's length in their shared representation in multi-modal models such as CLIP. Our systematic analysis demonstrates that this gap is caused by a combination of model initialization and contrastive learning optimization. In model initialization, we show empirically and theoretically that the representation of a common deep neural network is restricted to a narrow cone. As a consequence, in a multi-modal model with two encoders, the representations of the two modalities are clearly apart when the model is initialized. During optimization, contrastive learning keeps the different modalities separate by a certain distance, which is influenced by the temperature parameter in the loss function. Our experiments further demonstrate that varying the modality gap distance has a significant impact in improving the model's downstream zero-shot classification performance and fairness. Our code and data are available at https://modalitygap.readthedocs.io/

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/32e89d4d1225f3e02ac38ec93efc39fb61ccdde8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Changyou Chen
2. Yufan Zhou
3. Jianfeng Gao
4. Jinhui Xu
5. Peng Jin
6. Jiaxi Cui
7. Munan Ning
8. Junwu Zhang

In Computer Vision and Pattern Recognition

1. Changyou Chen
2. Yufan Zhou
3. Jinhui Xu
4. Bingchen Liu
5. Yizhe Zhu
6. Jianfeng Gao
7. Peng Jin
8. Jiaxi Cui

Citations in the corpus

• Lafite2: Few-shot Text-to-Image Generation (2022, background): “A recent work [18] also reveals a related phenomenon in the feature spaces of multi-modal models like CLIP: features from different encoders concentrate on different narrow cones of the feature space, and different modalities are actually not well-aligned in the multi-modal feature space.”
• DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, neutral): “However, CLIP models primarily learn modality-shared semantics, often overlooking modality-specific information due to a modality gap (Liang et al., 2022; Liu et al., 2023f).”
• Separate Anything You Describe (2023, neutral): “Overall,weob-servedthatusingadditionalaudiosupervisiontoscaleAudioSep didnotleadtoimprovementsinourexperiments,likelydueto the modality gap phenomenon [81]: the embedding spaces in the CLAP models may not be well aligned.”
• Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models (2023, neutral): “This ineffectiveness arises because, in CLIP’s joint vision-language embedding space, the embedding vectors of images and text are distinctly separated—a phenomenon referred to as the “Modality Gap” by Liang et al. (2022).”
• Shifted Diffusion for Text-to-image Generation (2022, methodology): “The reason is that, as revealed in [11], the effective output space of the CLIP image encoder is actually a very small region of the whole embedding space, as shown in Figure 3.”
• MoE-LLaVA: Mixture of Experts for Large Vision-Language Models (2024, background): “This is because multi-modal data naturally exhibit gaps (Liang et al., 2022), making it difficult for soft routers to learn the optimal patterns for assigning tokens to different experts.”
• Nomic Embed Vision: Expanding the Latent Space (2024, unknown): “Liang et al. (2022) and not aligned in this work.”
• When Embedding Models Meet: Procrustes Bounds and Applications (2025, unknown): “However, these models exhibit a persistent modality gap : embeddings cluster by modality in disjoint regions of R D (Liang et al., 2022).”
• Query Drift Compensation: Enabling Compatibility in Continual Learning of Retrieval Embedding Models (2025, unknown): “In the case of multi-modal retrieval (Wang et al., 2021), the unequal drift arises from the modality gap at initialization which is preserved during the contrastive training approach (as discussed in Liang et al. (2022) for vision-language models).”