Ferret: Refer and Ground Anything Anywhere at Any Granularity

By Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, Yinfei Yang. International Conference on Learning Representations, 2023

563 citations recorded by the source corpus.

We introduce Ferret, a new Multimodal Large Language Model (MLLM) capable of understanding spatial referring of any shape or granularity within an image and accurately grounding open-vocabulary descriptions. To unify referring and grounding in the LLM paradigm, Ferret employs a novel and powerful hybrid region representation that integrates discrete coordinates and continuous features jointly to represent a region in the image. To extract the continuous features of versatile regions, we propose a spatial-aware visual sampler, adept at handling varying sparsity across different shapes. Consequently, Ferret can accept diverse region inputs, such as points, bounding boxes, and free-form shapes. To bolster the desired capability of Ferret, we curate GRIT, a comprehensive refer-and-ground instruction tuning dataset including 1.1M samples that contain rich hierarchical spatial knowledge, with 95K hard negative data to promote model robustness. The resulting model not only achieves superior performance in classical referring and grounding tasks, but also greatly outperforms existing MLLMs in region-based and localization-demanded multimodal chatting. Our evaluations also reveal a significantly improved capability of describing image details and a remarkable alleviation in object hallucination. Code and data will be available at https://github.com/apple/ml-ferret

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/458111ac5a0f73bb35a2acf55298268be25ccfa2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, methodology): “In contrast, VisionLLM [27], GPT4RoI [28], [29], and Ferret [30, 31], along with GLaMM [32], use spatial boxes with ROI-aligned features to map region-level features into the LLM word embedding space.”
A survey on multimodal large language models (2023, unknown): “In order to further inspect what roles LLMs exactly play in LLM-Aided Visual Reasoning systems, existing related works are divided into three types: • LLM as a Controller • LLM as a Decision Maker”
GPT-4V(ision) is a Generalist Web Agent, if Grounded (2024, background): “While all these prior works show promise, generalizing to various web environments remains a challenge for existing models.”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, background): “On referring expression, U NIFIED -IO 2 is ahead of Shikra [26] and Ferret [207] and matches the scores achieved by Qwen-VL.”
GLaMM: Pixel Grounding Large Multimodal Model (2023, methodology): “Recent works like Kosmos-2 [35], Shikra [5], GPT4RoI [57], VisionLLM [44], Ferret [53] and All-Seeing [45] aim to allow region-specific conversation.”
Generative Multimodal Models are In-Context Learners (2023, background): “A school of successive efforts [69, 76, 84, 90, 91] further improves visual instruction tuning with better overall training pipelines [6, 40], grounding annotations [14, 15, 85, 92], and extra tasks [6].”
A Survey on Hallucination in Large Vision-Language Models (2024, background): “Another valuable dataset is GRIT [You et al. , 2023].”
CogVLM: Visual Expert for Pretrained Language Models (2023, unknown): “We compared with the latest state-of-the-art generalist models, including MiniGPT-4 [Zhu et al., 2023], IDEFICS-Instruct [Laurençon et al., 2023], OpenFlamingo [Awadalla et al., 2023], DreamLLM [Dong et al., 2023], InstructBLIP [Dai et al., 2023], Fuyu [Bavishi et al., 2023], Qwen-VL [Bai et al.,…”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “…Data [129], ScienceQA [165], AI2D [109], Knowledge TQA [110], Wikipedia-QA [81], Synthetic Multidisciplinary Knowledge / QA Objects365 [208], GRIT [278], RefCOCO [280], GPT4Gen-RD-BoxCoT [27], All-Seeing-V1 [251], Grounding All-Seeing-V2 [250], V3Det [243], TolokaVQA [236] Document DocReason25K…”