Scaling Language-Image Pre-Training via Masking

By Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, Kaiming He. 2022

431 citations recorded by the source corpus.

We present Fast Language-Image Pre-training (FLIP), a simple and more efficient method for training CLIP [52]. Our method randomly masks out and removes a large portion of image patches during training. Masking allows us to learn from more image-text pairs given the same wall-clock time and contrast more samples per iteration with similar memory footprint. It leads to a favorable trade-off between accuracy and training time. In our experiments on 400 million image-text pairs, FLIP improves both accuracy and speed over the no-masking baseline. On a large diversity of downstream tasks, FLIP dominantly outperforms the CLIP counterparts trained on the same data. Facilitated by the speedup, we explore the scaling behavior of increasing the model size, data size, or training length, and report encouraging results and comparisons. We hope that our work will foster future research on scaling vision-language learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Improving CLIP Training with Language Rewrites (2023, methodology): “Follow-up methods that build upon language-image pre-training, such as SLIP [41] and FLIP [35], exhibit similar advantageous scaling performance.”
EVA-CLIP: Improved Training Techniques for CLIP at Scale (2023, background): “In this work, we leverage FLIP to improve the time efficiency of training CLIP models.”
Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, positive): “Speciﬁcally, Sun et al. [70] present a method to improve training stability and reduce computational costs, including improved initialization, better optimizer, and random masking of images [66].”
BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs (2023, methodology): “Additionally, it implements a patch dropout strategy [28] to maintain pretraining efficiency while enhancing model performance.”
Foundation models in robotics: Applications, challenges, and the future (2023, neutral): “FLIP [111] proposes a simple and more efﬁcient training method for CLIP.”
MOMENT: A Family of Open Time-series Foundation Models (2024, background): “Masked language (Devlin et al., 2019; Raffel et al., 2020) and image modeling (Xie et al., 2022; Li et al., 2023b) have been successfully utilized to learn models from vast quantities of unlabeled data, which can generalize to a variety of downstream tasks.”
GLAZE: Protecting Artists from Style Mimicry by Text-to-Image Models (2023, background): “More powerful models continue to arrive [34, 57, 83], catalyzed by VC funding [59, 97, 98], technical research breakthroughs [4, 11, 37, 46, 52], and powered at their core by continuous training on a large volume of human-made art scraped from online art repositories such as ArtStation, Pinterest an”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, methodology): “• In addition, since the diffusion model is used as the image decoder, it would be interesting to explore DMs-based zero-shot perception such as classification (Li et al., 2023a) and segmentation (Xu et al., 2023a) with the goal of unifying all perception tasks with image synthesis/generation.”
An Image is Worth 32 Tokens for Reconstruction and Generation (2024, neutral): “This strategy has been validated across a variety of computer vision tasks that rely on high-level image features [30, 41].”
Transformer-Based Visual Segmentation: A Survey (2023, background): “Recent work [157] demonstrates the effectiveness of VLM in downstream tasks, including IS and SS.”
RemoteCLIP: A Vision Language Foundation Model for Remote Sensing (2023, background): “Recent advancements in CLIP models have primarily concentrated on scaling the model size and data size [53], incorporating self-supervision [54], [55], [56], enhancing pretraining efficiency [57], [58], and few-shot adaptation [59], [60], among others.”
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (2023, background): “The results show that a mask ratio of 0.5 demonstrates the highest performance, requiring only a quarter of the computational resources, aligning with findings in FLIP (Li et al., 2023b).”