ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models

By Haotian Liu, Liunian Harold Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Yong Jae Lee, Houdong Hu, Zicheng Liu, Jianfeng Gao. 2022

195 citations recorded by the source corpus.

Learning visual representations from natural language supervision has recently shown great promise in a number of pioneering works. In general, these language-augmented visual models demonstrate strong transferability to a variety of datasets and tasks. However, it remains challenging to evaluate the transferablity of these models due to the lack of easy-to-use evaluation toolkits and public benchmarks. To tackle this, we build ELEVATER (Evaluation of Language-augmented Visual Task-level Transfer), the first benchmark and toolkit for evaluating(pre-trained) language-augmented visual models. ELEVATER is composed of three components. (i) Datasets. As downstream evaluation suites, it consists of 20 image classification datasets and 35 object detection datasets, each of which is augmented with external knowledge. (ii) Toolkit. An automatic hyper-parameter tuning toolkit is developed to facilitate model evaluation on downstream tasks. (iii) Metrics. A variety of evaluation metrics are used to measure sample-efficiency (zero-shot and few-shot) and parameter-efficiency (linear probing and full model fine-tuning). ELEVATER is a platform for Computer Vision in the Wild (CVinW), and is publicly released at at https://computer-vision-in-the-wild.github.io/ELEVATER/

Subfields: Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Jianfeng Gao
4. Jianfeng Wang
5. Xiaowei Hu
6. Linjie Li
7. Chunyuan Li
8. Jianwei Yang

In Evaluation & Benchmarks

1. Jianwei Yang
2. Jianfeng Gao
3. Yong Jae Lee
4. Zi-Yi Dou
5. Zhe Gan
6. Lu Yuan
7. Siyi Liu
8. Yuheng Li

Citations in the corpus

• Visual Instruction Tuning (2023, background): “One of the core aspirations in artificial intelligence is to develop a general-purpose assistant that can effectively follow multi-modal vision-and-language instructions, aligned with human intent to complete various real-world tasks in the wild [4, 24].”
• A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, methodology): “To further study the object detection ability of OpenSeeD , we follow GLIP [30] to evaluate detection performance on Object Detection in the Wild (ODinW) benchmark [27].”
• Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, background): “ODinW Benchmark ODinW (Object Detection in the Wild) [23] is a more challenging benchmark to test model performance under real-world scenarios.”
• GLIGEN: Open-Set Grounded Text-to-Image Generation (2023, methodology): “To enable our model to ground open-world vocabulary concepts [29,31,69,72], we use the same pre-trained text encoder (for encoding the caption) to encode each phrase associated with each grounded entity (i.”
• Generalized Decoding for Pixel, Image, and Language (2022, methodology): “[42] Chunyuan Li, Haotian Liu, Liunian Harold Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Yong Jae Lee, Houdong Hu, Zicheng Liu, et al.”
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters (2021, neutral): “In Table 4, we compare CLIP-Adapter with the newly proposed ELEVATER benchmark [34] for pretrained vision-language models.”
• OCRBench: on the hidden mystery of OCR in large multimodal models (2023, unknown): “The success of large models has also been extended to the multimodal vision-language space [7], leading to a line of research on large multimodal models (LMM), including contrastive learning [8, 9, 10, 11] and generative modeling [2, 12, 13, 14, 15].”
• LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day (2023, background): “…inputs, the resulting large multimodal models (LMMs) exhibit strong zero-shot task completion performance on a variety of user-oriented vision-language tasks such as image understanding and reasoning, paving the way to develop general-purpose multimodal conversational assistants [22, 2, 23, 10].”
• Learning Customized Visual Models with Retrieval-Augmented Knowledge (2023, methodology): “We then further evaluate our model on ELEVATER [27], which is an open-set image classification benchmark that contains 20 datasets.”
• K-LITE: Learning Transferable Visual Models with External Knowledge (2022, methodology): “We apply the proposed knowledge-augmented models to two task-level transfer settings defined in ELEVATER benchmark [47], which evaluates the transferability of the learned visual representations in the wild.”
• Scaling Open-Vocabulary Object Detection (2023, methodology): “Self-training yields continued improvements on a suite of diverse datasets (ODinW13; x -axis), but performance on any given dataset (e.g. LVIS; y -axis) may saturate (red circles).”
• Focal Modulation Networks (2022, methodology): “0 Table 5: Zero-shot performance comparison of FocalNet-B and Swin-B on ELEVATER benchmark [42].”
• Vision-Language Models for Vision Tasks: A Survey (2023, unknown): “Object Detection COCO 2014 Detection [128] [link] 2014 80 83,000 41,000 box mAP COCO 2017 Detection [128] [link] 2017 80 118,000 5,000 box mAP LVIS [129] [link] 2019 1203 118,000 5,000 box mAP ODinW [130] [link] 2022 314 132413 20070 box mAP”
• BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs (2023, methodology): “In general domains, comprehensive benchmarks like ELEVATER [29] have spurred rapid advances in vision-language pretraining by facilitating direct comparison among pretrained models.”