Learning Transferable Visual Models From Natural Language Supervision
By Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Gabriel Goh, S. Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin and 3 more. International Conference on Machine Learning, 2021
52,559 citations recorded by the source corpus.
State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained model weights at https://github.com/OpenAI/CLIP.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning
View Paper: https://arxiv.org/abs/2103.00020
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Sergey Levine
- Percy Liang
- Quoc V. Le
- Luke S. Zettlemoyer
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Andrea Vedaldi
In Multimodal Learning
In Representation & Self-Supervised Learning
- Trevor Darrell
- Andrea Vedaldi
- Yoshua Bengio
- Andrew Zisserman
- Ishan Misra
- Dacheng Tao
- Sergey Levine
- Quoc V. Le
Citations in the corpus
- High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “[19] we compute FID, Inception and CLIP [60] scores for the Text-to-Image models from Tab.”
- Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, methodology): “Recent progress in computer vision has been driven by scaling models on large datasets of captioned images collected from the internet [10, 44, 60, 39, 31, 16].”
- Evaluating Large Language Models Trained on Code (2021, methodology): “Generative models have been shown to encode bias in modalities such as natural language (Brown et al., 2020; Blodgett et al., 2020) and images (Radford et al., 2021), and we find that the same is true of models like Codex that generate code.”
- PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation (2021, background): “also expected for relieving the computational cost of model training and inference. • Pretrained multi-modal models integrating language, vision and speech data have attracted much attention recently [42, 16]. Similar to the scaling law of language models, the performance of pertrained multi-modal m”
- The Curse of Recursion: Training on Generated Data Makes Models Forget (2023, background): “But with the emergence of contrastive learning [Radford et al., 2021] and LLMs [Brown et al.”
- A Review of Sparse Expert Models in Deep Learning (2022, methodology): “The model outperforms CLIP when using a comparable training strategy, and when scaled further matches state-of-the-art methods.”
- Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, methodology): “Since its original release in 2020, ConVIRT has directly inspired subsequent studies such as the CLIP framework (Radford et al., 2021) and the ALIGN model (Jia et al., 2021), which showed that direct adaptations of ConVIRT-style pretraining at much larger scales lead to state-of-the-art general…”
- Do Image Classifiers Generalize Across Time? (2019, background): “Recently introduced, contrastive models trained on weakly supervised web images [23] can reduce this gap, but require over 400 million images, and still exhibit noticeable gaps of 6.”
- An Empirical Study of Training Self-Supervised Vision Transformers (2021, methodology): “By default, we use AdamW [30] as the optimizer, which is the common choice for training ViT models [15, 41, 35].”
- Zero-Shot Text-to-Image Generation (2021, methodology): “Similar to Razavi et al. (2019), we rerank the samples drawn from the transformer using a pretrained contrastive model (Radford et al., 2021).”
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “While some methods rely on image re-ranking for post-generation image filtering (utilizing CLIP [44] for instance), we extend the use of classifier-free guidance suggested for diffusion models [53, 20] by [22, 41] to transformers, eliminating the need for post-generation filtering, thus producing…”
- GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, methodology): “For CLIP score, we employ the CLIP ViT-B/16 model released by Radford et al. (2021), and scale scores by the CLIP logit scale (100 in this case).”
- Generative Adversarial Networks (2021, neutral): “GANs have emerged as a useful technique for producing realistic data in a variety of disciplines ranging from computer vision and graphics to NLP and audio synthesis.”
- Towards Language-Free Training for Text-to-Image Generation (2021, methodology): “This idea is recently scaled up in CLIP/ALIGN [15,35], which pave the way toward building a universal image-text representation space.”
- LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs (2021, methodology): “For generation, we use CLIP ViT-B/16 [1] to rank the top 8 of 128 total samples per caption.”
- Projected GANs Converge Faster (2021, methodology): “To analyze the dependency on ImageNet features (Section 4.3), we also consider R50-CLIP [60], a ResNet optimized with a contrastive language-image objective on a dataset of 400 million (image, text) pairs.”
- ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis (2021, methodology): “2, we evaluate FID [25] and Inception Scores (IS) [58] to measure the quality of synthesized images, as well as cosine similarity between CLIP [52] embeddings of the text prompts and the synthesized images to measure how well the image reflects the text.”
- CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders (2021, methodology): “The recent introduction of CLIP (Radford et al., 2021), a dual language-image encoder, is a large step towards unifying textual and visual information.”
- D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, neutral): “, DALL-E [74] and CLIP [73] over image-text pairs).”