LiT: Zero-Shot Transfer with Locked-image text Tuning

By Xiaohua Zhai, Xiao Wang, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, Lucas Beyer. 2021

757 citations recorded by the source corpus.

This paper presents contrastive-tuning, a simple method employing contrastive training to align image and text mod-els while still taking advantage of their pre-training. In our empirical study we find that locked pre-trained image mod-els with unlocked text models work best. We call this in-stance of contrastive-tuning “Locked-image Tuning” (LiT), which just teaches a text model to read out good repre-sentations from a pre-trained image model for new tasks. A LiT model gains the capability of zero-shot transfer to new vision tasks, such as image classification or retrieval. The proposed LiT is widely applicable; it works reliably with multiple pre-training methods (supervised and unsu-pervised) and across diverse architectures (ResNet, Vision Transformers and MLP-Mixer) using three different image-text datasets. With the transformer-based pre-trained ViT-g/14 model, the LiT model achieves 84.5% zero-shot trans-fer accuracy on the ImageNet test set, and 81.1% on the challenging out-of-distribution ObjectNet test set.

Subfields: Artificial Intelligence, Multimodal Learning, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Multimodal Learning

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “1) it seems likely we may be able to improve our performance by switching to ViT-B/16, or other recent more-performant VLM models (Zhai et al. 2021).”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, methodology): “Finally, state-of-the-art zero-shot models on ImageNet such as BASIC [82] and LiT [146] are particularly optimized on classification tasks as they are trained on JFT-3B [145], a dataset with images and labels.”
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation (2022, methodology): “Dual-stream models (Lu et al., 2019; Radford et al., 2021; Jia et al., 2021; Zhai et al., 2021; Yao et al., 2022) use separate encoders for images and texts, allowing efficient inference for downstream multimodal alignment tasks like image-text retrieval, by pre-computing image/text features offline”
Combined Scaling for Zero-shot Transfer Learning (2021, background): “Zero-shot transfer learning Instead of zero-shot learning, CLIP and ALIGN are known to conduct zero-shot transfer learning (Radford et al., 2021; Jia et al., 2021; Zhai et al., 2022).”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “The current working directory is /home/user/ echo “I am a chatbot“ > file.txt && cat file.txt && ls -l I am a chatbot -rw-r–r– 1 user user”
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, methodology): “To fuse the vision processing ability into such LLMs, several challenges are lying ahead since it is hard to train either large language models or vision models, and the well-designed instructions [4, 55, 21] and cumbersome conversions [30, 52] are required to connect different modalities.”
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023, methodology): “…(Radford et al., 2021; Jia et al., 2021), the fusion-encoder architecture (Tan & Bansal, 2019; Li et al., 2021), the encoder-decoder architecture (Cho et al., 2021; Wang et al., 2021b; Chen et al., 2022b), and more recently, the uniﬁed transformer architecture (Li et al., 2022; Wang et al., 2022b).”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, methodology): “We follow [94] and evaluate robustness performance on ImageNet distribution shift datasets [3, 23, 25, 61, 82].”
CoCa: Contrastive Captioners are Image-Text Foundation Models (2022, methodology): “To further improve zero-shot image classification accuracy, LiT [32] and BASIC [33] first pretrain model on an largescale image annotation dataset with cross-entropy and further finetune with contrastive loss on an noisy alt-text image dataset.”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “ViT-e has the same architecture and uses the same training recipe as the 1.8B parameter ViT-G model (Zhai et al., 2022a), while scaling to 4B parameters.”
Robust fine-tuning of zero-shot models (2021, background): “NS EfﬁcientNet-L2 (Xie et al., 2020) 88.4 80.2 74.7 47.6 68.5 84.9 71.2 79.8 ViT-G/14 (Zhai et al., 2021) 90.4 83.3 - - 70.5 - - - Zero-shot ALIGN (Jia et al., 2021) 76 ularization term (e.g., weight decay).”
Simple Open-Vocabulary Object Detection with Vision Transformers (2022, methodology): “By leveraging large amounts of image-text data, contrastive training has yielded major improvements in zero-shot classification performance and other language-based tasks [33,19,44].”
From Sparse to Soft Mixtures of Experts (2023, methodology): “Following Zhai et al. (2022b), the image tower is pre-trained on image classification, and then frozen while training the text encoder on a dataset of image-text pairs.”
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought (2023, methodology): “For example, models like Uniter [18], Oscar [19], VinVL [20], and LiT [21] freeze the image encoder, while Frozen [22] and VGPT [23] freeze the language model.”
MaPLe: Multi-modal Prompt Learning (2022, methodology): “Recently, V-L models like CLIP [32], ALIGN [15], LiT [45], FILIP [41] and Florence [43] have demonstrated exceptional performance on a wide spectrum of tasks including few-shot and zero-shot visual recognition.”
Vision-Language Models for Vision Tasks: A Survey (2023, background): “LiT [143] [project] CC12M, YFCC100M, WIT∗ Con Propose contrastive tuning with the locked image encoder.”
Scaling Vision Transformers to 22 Billion Parameters (2023, background): “Table 3 shows the zero-shot transfer results of ViT-22B against CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021), BASIC (Pham et al., 2021), CoCa (Yu et al., 2022a), LiT (Zhai et al., 2022b) with ViT-g (Zhai et al., 2022a) and ViT-e (Chen et al., 2022) (Taori et al., 2020).”
A Cookbook of Self-Supervised Learning (2023, methodology): “When it is not explicitly needed by the method, using the average of the patch tokens instead of the class token saves memory without much change on the accuracies of the network [Zhai et al., 2022a].”
ImageBind One Embedding Space to Bind Them All (2023, background): “LiT [82] adopts contrastive training for fine-tuning and observes freezing image encoders works the best.”