LISA: Reasoning Segmentation via Large Language Model

By Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, Jiaya Jia. 2023

972 citations recorded by the source corpus.

Although perception systems have made remarkable ad-vancements in recent years, they still rely on explicit human instruction or pre-defined categories to identify the target objects before executing visual recognition tasks. Such systems cannot actively reason and comprehend implicit user intention. In this work, we propose a new segmentation task - reasoning segmentation. The task is designed to output a segmentation mask given a complex and implicit query text. Furthermore, we establish a benchmark comprising over one thousand image-instruction-mask data samples, incorporating intricate reasoning and world knowledge for evaluation purposes. Finally, we present LISA: large Language Instructed Segmentation Assistant, which inherits the language generation capabilities of multimodal Large Language Models (LLMs) while also possessing the ability to produce segmentation masks. We expand the original vocabulary with a token and propose the embedding-as-mask paradigm to unlock the segmentation capability. Remarkably, LISA can handle cases involving complex rea-soning and world knowledge. Also, it demonstrates robust zero-shot capability when trained exclusively on reasoning-free datasets. In addition, fine-tuning the model with merely 239 reasoning segmentation data samples results in further performance enhancement. Both quantitative and qualitative experiments show our method effectively unlocks new reasoning segmentation capabilities for multimodal LLMs. Code, models, and data are available at github.com/dvlab-research/LISA.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

Citations in the corpus

Improved Baselines with Visual Instruction Tuning (2023, background): “The LLaVA architecture is also lever-aged in different downstream tasks and domains, including region-level [8, 55] and pixel-level [25, 49] understanding, biomedical assistants [30], image generation [5], adversarial studies [6, 58].”
VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model (2025, positive): “In the context of REC, our 3B RL model achieves a score of 63.16 on the out-of-domain evaluation benchmark LISA-Grounding [24](vs. 54.82 for SFT).”
V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs (2023, background): “In order to localize and crop the searched targets for VWM, it’s also necessary to enhance the MLLM with additional localization capabilities, comparable to those mentioned in [20, 58].”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, background): “Recent advancements have seen the creation of vision large language models (VLLMs) [3, 21, 56, 59, 62, 66, 100, 121, 128, 130, 136, 138, 141, 142, 148], which aim to enhance language models with the capability to process and interpret visual information.”
Visual-RFT: Visual Reinforcement Fine-Tuning (2025, positive): “In reasoning grounding, Visual-RFT excels in the LISA [13] dataset, which heavily relies on reasoning, outperforming specialized models like GroundedSAM [20].”
OmniGen: Unified Image Generation (2024, neutral): “Additionally, we also use several datasets for referring image segmentation, including Ref-COCO [26], ADE20k [78], and ReasonSeg [29].”
LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models (2023, background): “The advancements in computer vision and NLP have led to the emergence of vision-language models (VLMs) that integrate vision models with language models for cross-modality understanding [11, 58] and reasoning [19, 27, 37].”
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (2024, background): “MLLMs have also been extended to various applications, including fine-grained localization [61, 24] such as object detection [72], video understanding [71, 29, 8], and image generation [20, 49].”
OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation (2023, neutral): “_Additionally , _there _are _two _cars _park ed _in _the MLLM allows it to be adept at a variety of vision tasks [2, 25, 45], meanwhile easily handling some complex content comprehension [23] or generation [4, 15].”
How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “The development of open-source MLLMs [4, 15, 20, 43, 47, 49, 52, 65, 66, 98, 105, 113, 115, 119, 133, 134] has significantly influenced the AGI landscape by integrating and enhancing capabilities in processing both visual and textual data.”
GLaMM: Pixel Grounding Large Multimodal Model (2023, methodology): “In contrast, LISA [21] utilizes embeddings from the vision language model and the SAM [18] decoder to generate output segmentation masks.”