GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest

By Shilong Zhang, Pei Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, Ping Luo. 2023

363 citations recorded by the source corpus.

Visual instruction tuning large language model(LLM) on image-text pairs has achieved general-purpose vision-language abilities. However, the lack of region-text pairs limits their advancements to fine-grained multimodal understanding. In this paper, we propose spatial instruction tuning, which introduces the reference to the region-of-interest(RoI) in the instruction. Before sending to LLM, the reference is replaced by RoI features and interleaved with language embeddings as a sequence. Our model GPT4RoI, trained on 7 region-text pair datasets, brings an unprecedented interactive and conversational experience compared to previous image-level models. (1) Interaction beyond language: Users can interact with our model by both language and drawing bounding boxes to flexibly adjust the referring granularity. (2) Versatile multimodal abilities: A variety of attribute information within each RoI can be mined by GPT4RoI, e.g., color, shape, material, action, etc. Furthermore, it can reason about multiple RoIs based on common sense. On the Visual Commonsense Reasoning(VCR) dataset, GPT4RoI achieves a remarkable accuracy of 81.6%, surpassing all existing models by a significant margin (the second place is 75.6%) and almost reaching human-level performance of 85.0%. The code and model can be found at https://github.com/jshilong/GPT4RoI.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

Improved Baselines with Visual Instruction Tuning (2023, neutral): “The LLaVA architecture is also lever-aged in different downstream tasks and domains, including region-level [8, 55] and pixel-level [25, 49] understanding, biomedical assistants [30], image generation [5], adversarial studies [6, 58].”
A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “In the realm of Model Compression, Quantization (Frantar et al., 2023; Dettmers et al., 2022; Kim et al., 2023c; Tao et al., 2022b; Yao et al., 2022; Xiao et al., 2023), Parameter Pruning (Ma et al., 2023d; Zhang et al., 2023h; Frantar and Alistarh, 2023), and Low-Rank Approximation (Xu et al.,…”
GLaMM: Pixel Grounding Large Multimodal Model (2023, methodology): “The last category combines specialized vision modules with LMMs, trained end-to-end for a comprehensive understanding of regions [21, 45, 57].”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, background): “This includes creating models that can do OCR [12, 220], visual grounding [12, 26, 143, 189, 207, 212, 219], image-text-retrieval [97], additional languages [112], embodied AI tasks [17, 135, 140, 152] or leverage other expert systems [52].”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, background): “More recently, some approaches have been developed towards visual-interactive multimodal comprehension by precise referring instruction tuning (Zhao et al., 2023a; Peng et al., 2023; Chen et al., 2023a; Zhang et al., 2023g).”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, background): “Recent advancements have seen the creation of vision large language models (VLLMs) [3, 21, 56, 59, 62, 66, 100, 121, 128, 130, 136, 138, 141, 142, 148], which aim to enhance language models with the capability to process and interpret visual information.”
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (2024, background): “MLLMs have also been extended to various applications, including fine-grained localization [61, 24] such as object detection [72], video understanding [71, 29, 8], and image generation [20, 49].”
Generative Multimodal Models are In-Context Learners (2023, background): “A school of successive efforts [69, 76, 84, 90, 91] further improves visual instruction tuning with better overall training pipelines [6, 40], grounding annotations [14, 15, 85, 92], and extra tasks [6].”
Hallucination of Multimodal Large Language Models: A Survey (2024, neutral): “This property results in incomplete cross-modal alignment and deprives the model of grounding ability [62, 126].”
Ferret: Refer and Ground Anything Anywhere at Any Granularity (2023, background): “Additional works in this direction include GPT4ROI (Zhang et al., 2023), PVIT (Chen et al., 2023a), BuboGPT (Zhao et al., 2023), VisionLLM (Wang et al., 2023), and ContextDET (Zang et al., 2023).”
A survey on multimodal large language models (2023, neutral): “In order to further inspect what roles LLMs exactly play in LLM-Aided Visual Reasoning systems, existing related works are divided into three types: • LLM as a Controller • LLM as a Decision Maker”
SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, methodology): “In contrast, VisionLLM [27], GPT4RoI [28], [29], and Ferret [30, 31], along with GLaMM [32], use spatial boxes with ROI-aligned features to map region-level features into the LLM word embedding space.”
How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “The development of open-source MLLMs [4, 15, 20, 43, 47, 49, 52, 65, 66, 98, 105, 113, 115, 119, 133, 134] has significantly influenced the AGI landscape by integrating and enhancing capabilities in processing both visual and textual data.”
LLaVA-Prumerge: Adaptive Token Reduction for Efficient Large Multimodal Models (2024, neutral): “More recent works extend whole-image understanding into region-level understanding [6, 7, 36, 54], video understanding [26, 53] and 3D scene understanding [18].”
OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation (2023, neutral): “_Additionally , _there _are _two _cars _park ed _in _the MLLM allows it to be adept at a variety of vision tasks [2, 25, 45], meanwhile easily handling some complex content comprehension [23] or generation [4, 15].”
LISA: Reasoning Segmentation via Large Language Model (2023, background): “GPT4RoI [57] introduces spatial boxes as input and trains the model on region-text pairs.”