Combined Scaling for Zero-shot Transfer Learning

By Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingxing Tan, Quoc V. Le. 2021

248 citations recorded by the source corpus.

We present a combined scaling method - named BASIC - that achieves 85.7% top-1 accuracy on the ImageNet ILSVRC-2012 validation set without learning from any labeled ImageNet example. This accuracy surpasses best published similar models - CLIP and ALIGN - by 9.3%. Our BASIC model also shows significant improvements in robustness benchmarks. For instance, on 5 test sets with natural distribution shifts such as ImageNet-{A,R,V2,Sketch} and ObjectNet, our model achieves 84.3% top-1 average accuracy, only a small drop from its original ImageNet accuracy. To achieve these results, we scale up the contrastive learning framework of CLIP and ALIGN in three dimensions: data size, model size, and batch size. Our dataset has 6.6B noisy image-text pairs, which is 4x larger than ALIGN, and 16x larger than CLIP. Our largest model has 3B weights, which is 3.75x larger in parameters and 8x larger in FLOPs than ALIGN and CLIP. Finally, our batch size is 65536 which is 2x more than CLIP and 4x more than ALIGN. We encountered two main challenges with the scaling rules of BASIC. First, the main challenge with implementing the combined scaling rules of BASIC is the limited memory of accelerators, such as GPUs and TPUs. To overcome the memory limit, we propose two simple methods which make use of gradient checkpointing and model parallelism. Second, while increasing the dataset size and the model size has been the defacto method to improve the performance of deep learning models like BASIC, the effect of a large contrastive batch size on such contrastive-trained image-text models is not well-understood. To shed light on the benefits of large contrastive batch sizes, we develop a theoretical framework which shows that larger contrastive batch sizes lead to smaller generalization gaps for image-text models such as BASIC.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Representation & Self-Supervised Learning

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, methodology): “Second, the classification performance of Flamingo lags behind that of state-of-the-art contrastive models [82, 85].”
Stable and low-precision training for large-scale vision-language models (2023, methodology): “As CLIP training requires large batch sizes [46, 44], this inner dimension of batch size times sequence length is much larger than for the other matrix multiplies.”
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “, 2021), and BASIC (Pham et al., 2021) models pretrained with contrastive supervision from image-text pairs, a ViT-G/14 model pre-trained on JFT-3B (Zhai et al.”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, methodology): “After CLIP’s initial success, ALIGN and BASIC improved contrastive multimodal learning by increasing the training set size and the batch size used for training [28, 54].”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, background): “The follow up work (Pham et al., 2021; Zhai et al., 2022b) studies the impact of the training data and batch size in contrastive learning.”
Robust fine-tuning of zero-shot models (2021, methodology): “While our investigation centers around CLIP, we observe similar trends for other zero-shot models including ALIGN [45], BASIC [77], and a ViT model pre-trained on JFT [21].”
Simple Open-Vocabulary Object Detection with Vision Transformers (2022, background): “This vastly increases the available training data and has led to large improvements on zero-shot classification benchmarks [33,19,44,32].”
EVA-02: A Visual Representation for Neon Genesis (2023, background): “These efforts are driven by the belief that increasing the number of parameters, data, and compute budgets will ultimately result in improved performance [63, 142, 134, 93].”
Scaling Language-Image Pre-Training via Masking (2022, background): “, [37, 51]) have popularized learning visual representations with language supervision.”
Scaling Vision Transformers to 22 Billion Parameters (2023, background): “Table 3 shows the zero-shot transfer results of ViT-22B against CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021), BASIC (Pham et al., 2021), CoCa (Yu et al., 2022a), LiT (Zhai et al., 2022b) with ViT-g (Zhai et al., 2022a) and ViT-e (Chen et al., 2022) (Taori et al., 2020).”
Sigmoid Loss for Language Image Pre-Training (2023, background): “BASIC [33] and LAION [48] look at scaling batch-size but only go up to 16 k and 160 k respectively, by using many hundreds of chips, and for the former also mixing in a large private classiﬁcation dataset [33, 51].”
CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration (2022, methodology): “Zero-shot techniques have become increasingly popular in image classification [53, 33, 51], object detection [4, 54, 19, 38, 77, 42, 26], and semantic segmentation [6, 35, 32, 3, 14, 76].”
Scaling Open-Vocabulary Object Detection (2023, methodology): “For image-level tasks, large improvements have been achieved through contrastive pretraining of vision-language models, which is massively scalable because it can use naturally abundant weak supervision in the form of image-text pairs from the Web [24, 9, 23].”
Reproducible Scaling Laws for Contrastive Language-Image Learning (2022, methodology): “This may appear at first sight modest compared to results reported by BASIC (85.7% [54]), LiT (85.2% [84]) or CoCA (86.1% [80]).”
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale (2022, background): “CLIP (Contrastive Language-Image Pre-training) [48,50,72, 73] is a type of multi-modal foundation model that connects vision and language via contrastive image-text pre-training.”
Intra-agent speech permits zero-shot task acquisition (2022, background): “Similar to phenomena of zero-shot generalization in supervised classification and object detection [20, 21, 41], these studies have often relied on a large, diverse dataset providing supervision – where held-out tasks represent a novel recombination of parts of the training dataset (e.”
F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models (2022, background): “Recently, popular large VLMs scale up by training on billions of image-text pairs and acquire strong image-text representation by contrastive learning (Radford et al., 2021; Jia et al., 2021; Pham et al., 2021; Zhai et al., 2022).”
DataComp: In search of the next generation of multimodal datasets (2023, background): “In numerous areas of machine learning, larger datasets lead to better performance [80, 71, 65, 100, 60, 24, 15, 104, 105].”
Editing Models with Task Arithmetic (2022, unknown): “[76] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingxing Tan, and Quoc V Le.”