K-LITE: Learning Transferable Visual Models with External Knowledge

By Sheng Shen, Chunyuan Li, Xiaowei Hu, Yujia Xie, Jianwei Yang, Pengchuan Zhang, Anna Rohrbach, Zhe Gan, Lijuan Wang, Lu Yuan and 4 more. 2022

107 citations recorded by the source corpus.

The new generation of state-of-the-art computer vision systems are trained from natural language supervision, ranging from simple object category names to descriptive captions. This form of supervision ensures high generality and usability of the learned visual models, due to the broad concept coverage achieved via large-scale data collection process. Alternatively, we argue that learning with external knowledge is a promising way which leverages a much more structured source of supervision and offers sample efficiency. We propose K-LITE, a simple strategy to leverage external knowledge for building transferable visual systems: In training, it enriches entities in text with WordNet and Wiktionary knowledge, leading to an efficient and scalable approach to learning image representations that uses knowledge about the visual concepts. In evaluation, the text is also augmented with external knowledge and then used to reference learned visual concepts (or describe new ones) to enable zero-shot and few-shot transfer of the pre-trained models. We study the performance of K-LITE on two important computer vision problems, image classification and object detection, benchmarking on 20 and 13 different existing datasets, respectively. The proposed knowledge-augmented models show significant improvement in transfer learning performance over existing methods. Our code is available at https://github.com/microsoft/klite.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Knowledge Graphs & Representation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Knowledge Graphs & Representation

Jianfeng Gao

Citations in the corpus

ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, background): “Its pre-training setting is the same as UniCL Swin, but external knowledge such as Wiktionary is leveraged in model pre-training.”
BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “In CV, pre-trained models that learn from hundreds of millions of image-caption pairs have achieved top performance on image-to-text benchmarks [7–9], while delivering remarkably photo-realistic results in text-to-image tasks [10–14].”
Visual Classification via Description from Large Language Models (2022, background): “K-LITE (Shen et al., 2022) shows that such knowledge from WordNet and Wiktionary has the potential to enhance prompts for vision-language models.”
Doubly Right Object Recognition: A Why Prompt for Visual Rationales (2022, methodology): “External knowledge allows models to learn the right features and obtain better transferability [26, 45].”
Vision-Language Models for Vision Tasks: A Survey (2023, background): “K-lite [147] [code] CC3M, CC12M, YFCC100M Con Leverage auxiliary datasets for training transferable VLMs.”
MedCLIP: Contrastive Learning from Unpaired Medical Images and Text (2022, unknown): “Though remedies like data augmentation (Li et al., 2021) and knowledge graph (Shen et al., 2022) were proposed, the magnitude of used data is still far larger than medical data.”