VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use
By Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, L. Schimdt. 2023
111 citations recorded by the source corpus.
We introduce VisIT-Bench (Visual InsTruction Benchmark), a benchmark for evaluation of instruction-following vision-language models for real-world use. Our starting point is curating 70 'instruction families' that we envision instruction tuned vision-language models should be able to address. Extending beyond evaluations like VQAv2 and COCO, tasks range from basic recognition to game playing and creative generation. Following curation, our dataset comprises 592 test queries, each with a human-authored instruction-conditioned caption. These descriptions surface instruction-specific factors, e.g., for an instruction asking about the accessibility of a storefront for wheelchair users, the instruction-conditioned caption describes ramps/potential obstacles. These descriptions enable 1) collecting human-verified reference outputs for each instance; and 2) automatic evaluation of candidate multimodal generations using a text-only LLM, aligning with human judgment. We quantify quality gaps between models and references using both human and automatic evaluations; e.g., the top-performing instruction-following model wins against the GPT-4 reference in just 27% of the comparison. VisIT-Bench is dynamic to participate, practitioners simply submit their model's response on the project website; Data, code and leaderboard is available at visit-bench.github.io.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
In Multimodal Learning
In Evaluation & Benchmarks
Citations in the corpus
- Improved Baselines with Visual Instruction Tuning (2023, background): “As the seminal work of visual instruction tuning, LLaVA [35] showcases commendable proficiency in visual reasoning capabilities, surpassing even more recent models on diverse benchmarks [4, 54] for real-life visual instruction-following tasks.”
- LLM Evaluators Recognize and Favor Their Own Generations (2024, unknown): “The general tendency of LLMs to prefer their own generations was first recognized in the context of LLM-based benchmarks (Bitton et al., 2023a; Zheng et al., 2024; Bai et al., 2024).”
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, background): “Recent works (Bitton et al., 2023; Yu et al., 2023) evaluate instruction-following and reasoning capabilities, underscoring the growing importance of generative foundation models in practical applications.”