PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining

By Yuting Gao, Jinfeng Liu, Zihan Xu, Jinchao Zhang, Ke Li, Chunhua Shen. 2022

160 citations recorded by the source corpus.

Large-scale vision-language pre-training has achieved promising results on downstream tasks. Existing methods highly rely on the assumption that the image-text pairs crawled from the Internet are in perfect one-to-one correspondence. However, in real scenarios, this assumption can be difficult to hold: the text description, obtained by crawling the affiliated metadata of the image, often suffers from the semantic mismatch and the mutual compatibility. To address these issues, we introduce PyramidCLIP, which constructs an input pyramid with different semantic levels for each modality, and aligns visual elements and linguistic elements in the form of hierarchy via peer-level semantics alignment and cross-level relation alignment. Furthermore, we soften the loss of negative samples (unpaired samples) so as to weaken the strict constraint during the pre-training stage, thus mitigating the risk of forcing the model to distinguish compatible negative pairs. Experiments on five downstream tasks demonstrate the effectiveness of the proposed PyramidCLIP. In particular, with the same amount of 15 million pre-training image-text pairs, PyramidCLIP exceeds CLIP on ImageNet zero-shot classification top-1 accuracy by 10.6%/13.2%/10.0% with ResNet50/ViT-B32/ViT-B16 based image encoder respectively. When scaling to larger datasets, PyramidCLIP achieves the state-of-the-art results on several downstream tasks. In particular, the results of PyramidCLIP-ResNet50 trained on 143M image-text pairs surpass that of CLIP using 400M data on ImageNet zero-shot classification task, significantly improving the data efficiency of CLIP.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Molecular Biology, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chunyuan Li
2. Yong Jae Lee
3. Jianfeng Gao
4. Jianwei Yang
5. Haotian Liu
6. Ce Liu
7. Zhe Gan
8. Kilho Son

In Computer Vision and Pattern Recognition

1. Yong Jae Lee
2. Jianfeng Gao
3. Luc Van Gool
4. Chunyuan Li
5. Yi Yang
6. Dacheng Tao
7. Jianwei Yang
8. Trevor Darrell

In Molecular Biology

1. Jianwei Yang
2. Yong Jae Lee
3. Jianfeng Gao

In Multimodal Learning

1. Chunyuan Li
2. Jianwei Yang
3. Yong Jae Lee
4. Jianfeng Gao
5. Haotian Liu
6. Ce Liu
7. Sheng Jin
8. Guangyao Chen

Citations in the corpus

• Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, background): “17 pyramidCLIP [146] arXiv-2022 image-text CNN+Trans CS Hierarchical image-text contrastive learning -”
• Learning Customized Visual Models with Retrieval-Augmented Knowledge (2023, background): “There are an increasing number of studies to improve their generality from various modeling perspectives, including training objectives [11,13,14,38,65,68], scaling techniques [9,40,65], data efficiency [25, 28], and leveraging multilingual correlations [9, 19].”
• Vision-Language Models for Vision Tasks: A Survey (2023, background): “PyramidCLIP [116] constructs multiple semantic levels and performs both cross-level and peer-level contrastive learning for effective VLM pre-training.”