Learning Customized Visual Models with Retrieval-Augmented Knowledge

By Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jae Lee, Chunyuan Li. 2023

88 citations recorded by the source corpus.

Image-text contrastive learning models such as CLIP have demonstrated strong task transfer ability. The high generality and usability of these visual models is achieved via a web-scale data collection process to ensure broad concept coverage, followed by expensive pre-training to feed all the knowledge into model weights. Alternatively, we propose React,REtrieval-Augmented CusTomization, a framework to acquire the relevant web knowledge to build customized visual models for target domains. We retrieve the most relevant image-text pairs $(\thicksim3\%$ of CLIP pre-training data) from the web-scale database as external knowledge and propose to customize the model by only training new modularized blocks while freezing all the original weights. The effectiveness of Reactis demonstrated via extensive experiments on classification, retrieval, detection and segmentation tasks, including zero, few, and full-shot settings. Particularly, on the zero-shot classification task, compared with CLIP, it achieves up to 5.4% improvement on ImageNet and 3.7% on the Elevaterbenchmark (20 datasets).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jianfeng Gao
  2. Chunyuan Li
  3. Jianwei Yang
  4. Haotian Liu
  5. Hoifung Poon
  6. Zhe Gan
  7. Sheng Zhang
  8. Tristan Naumann

In Computer Vision and Pattern Recognition

  1. Chunyuan Li
  2. Jianfeng Gao
  3. Jianwei Yang
  4. Haotian Liu
  5. Luc Van Gool
  6. Xiaogang Wang
  7. Dacheng Tao
  8. Trevor Darrell

Citations in the corpus