Unified Contrastive Learning in Image-Text-Label Space

By Jianwei Yang, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, Jianfeng Gao. 2022

296 citations recorded by the source corpus.

Visual recognition is recently learned via either super-vised learning on human-annotated image-label data or language-image contrastive learning with webly-crawled image-text pairs. While supervised learning may result in a more discriminative representation, language-image pretraining shows unprecedented zero-shot recognition ca-pability, largely due to the different properties of data sources and learning objectives. In this work, we intro-duce a new formulation by combining the two data sources into a common image-text-label space. In this space, we propose a new learning paradigm, called Unified Con-trastive Learning (UniCL) with a single learning objective to seamlessly prompt the synergy of two data types. Ex-tensive experiments show that our UniCL is an effective way of learning semantically rich yet discriminative repre-sentations, universally for image recognition in zero-shot, linear-probing, fully finetuning and transfer learning sce-narios. Particularly, it attains gains up to 9.2% and 14.5% in average on zero-shot recognition benchmarks over the language-image contrastive learning and supervised learning methods, respectively. In linear probe setting, it also boosts the performance over the two methods by 7.3% and 3.4%, respectively. Our study also indicates that UniCL stand-alone is a good learner on pure image-label data, rivaling the supervised learning methods across three im-age classification datasets and two types of vision back-bones, ResNet and Swin Transformer. Code is available at: https://github.com/microsoft/UniCL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Visual Instruction Tuning (2023, background): “…foundation vision models [27, 16], with strong capabilities in open-world visual understanding such as classification [40, 21, 57, 54, 39], detection [29, 62, 33], segmentation [25, 63, 58] and captioning [50, 28], as well as visual generation and editing [42, 43, 56, 15, 44, 30].”
A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, methodology): “During fine-tuning, we use 512 × 1024 for Cityscapes [6] and 640 × 640 for ADE20K [58] by default.”
Generalized Decoding for Pixel, Image, and Language (2022, methodology): “In light of the versatility of transformers [72], we are now witnessing a growing interest in building general-purpose models that can learn from and be applied to a diverse set of vision and vision-language tasks, through multi-task learning [27, 32], sequential decoding [7, 54, 76, 85], or unified”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “Meanwhile, different test image requires different support shots based on Yang et al. (2022).”
Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation (2022, background): “We now discuss them within a single item because they can basically be formulated with the same objective [39, 47], and they possess some similar properties.”
InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, background): “Florence [15] extends them with uniﬁed contrastive learning [16] and elaborate adaptation models, which support a wide range of vision tasks in different transfer settings.”
GLIPv2: Unifying Localization and Vision-Language Understanding (2022, background): “The proposed loss can be viewed as a region-word level generalization of the prevalent image-level contrastive learning [33, 44, 55].”
Sigmoid Loss for Language Image Pre-Training (2023, background): “BASIC [28] and LAION [1] look at scaling batch-size but only go up to 16 k and 160 k respectively, by using many hundreds of chips, and for the former also mixing in a large private classification dataset [28, 41].”
TEMOS: Generating diverse human motions from textual descriptions (2022, background): “Others build joint cross-modal embeddings to map the text and motion to the same space [56,2,14], which has been a success in other research area [45,59,58,5].”
Learning Customized Visual Models with Retrieval-Augmented Knowledge (2023, methodology): “Note (4) is a general form; it reduces to UniCL [76] when γ = 1.”
Segment Everything Everywhere All at Once (2023, methodology): “For the language encoder, we adopt a UniCL or Florence text encoder [60, 61].”
Vision-Language Models for Vision Tasks: A Survey (2023, background): “This type of pre-training introduces image classification labels [65] into the image-text contrast as defined in Eq.”
DaViT: Dual Attention Vision Transformers (2022, methodology): “with unified contrastive learning [70,66] approach, DaViT-Huge and DaViTGiant reach 90.”