V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs

By Penghao Wu, Saining Xie. Computer Vision and Pattern Recognition, 2023

544 citations recorded by the source corpus.

When we look around and perform complex tasks, how we see and selectively process what we see is crucial. How-ever, the lack of this visual search mechanism in current multimodal LLMs (MLLMs) hinders their ability to focus on important visual details, especially when handling high-resolution and visually crowded images. To address this, we introduce V*, an LLM- guided visual search mechanism that employs the world knowledge in LLMs for efficient visual querying. When combined with an MLLM, this mechanism enhances collaborative reasoning, contextual understanding, and precise visual grounding. This integration results in a new MLLM meta-architecture, named Show, sEArch, and TelL (SEAL). We further create V* Bench, a benchmark specifically designed to evaluate MLLMs in their ability to process high-resolution images and focus on visual details. Our study highlights the necessity of incorporating visual search capabilities into multimodal systems. The code is available here.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c672ec79f55cef8f7a32cd8dddfa981b893f1567

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

MM-LLMs: Recent Advances in MultiModal Large Language Models (2024, unknown): “…et al., 2023b), mPLUG-PaperOwl( I D )(Hu et al., 2023a), RLHF-V (Yu et al., 2023b), Silkie (Li et al., 2023g), Lyrics (Lu et al., 2023a), VILA (Lin et al., 2023), CogAgent (Hong et al., 2023), Osprey (Yuan et al., 2023a), V* (Wu and Xie, 2023), MobileVLM (Chu et al., 2023a), TinyGPT-V (Yuan et al.)”
Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models (2024, methodology): “For computer vision, we tackle diverse tasks including (1) depth, (2) spatial reasoning, (3) jigsaw, (4) visual correspondence, (5) semantic correspondence, as well as questions from (6) the MMVP and (7) the V* Bench benchmarks [9, 44, 51].”
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (2024, background): “More recently, V ∗ [64] and CogCoM[48] modify the general mechanism in MLLMs and collect a series of visual reasoning steps as training data.”
MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024, background): “…models rely on superficial cues rather than true multimodal understanding (Du et al., 2023; Yuksekgonul et al., 2023), we risk overestimating their capabilities and potentially deploying systems that fail in unpredictable ways when faced with novel scenarios (Wu and Xie, 2024; Tong et al., 2024b).”
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, methodology): “For specialized tasks like V*Bench [127], which require processing ultra high-resolution images, increasing the resolution and number of visual tokens could lead to an HD version of the Cambrian-1 model.”
Qwen3-VL Technical Report (2025, unknown): “Notably, Qwen3-VL-235B-A22B achieves the state-of-the-art performance across all three benchmarks when augmented with tools—reaching 93.7 on V* (Wu &Xie, 2024), 85.3 on HRBench-4k (Wang et al., 2024e), and 82.3 on HRBench-8k (Wang et al., 2024e).”