GLaMM: Pixel Grounding Large Multimodal Model

By Hanoona Abdul Rasheed, Muhammad Maaz, Sahal Shaji Mullappilly, Abdelrahman M. Shaker, Salman Hameed Khan, Hisham Cholakkal, Rao Muhammad Anwer, Eric P. Xing, Ming-Hsuan Yang, Fahad Shahbaz Khan. Computer Vision and Pattern Recognition, 2023

537 citations recorded by the source corpus.

Large Multimodal Models (LMMs) extend Large Lan-guage Models to the vision domain. Initial LMMs used holistic images and text prompts to generate ungrounded textual responses. Recently, region-level LMMs have been used to generate visually grounded responses. However, they are limited to only referring to a single object category at a time, require users to specify the regions, or can-not offer dense pixel-wise object grounding. In this work, we present Grounding LMM (GLaMM), the first model that can generate natural language responses seamlessly in-tertwined with corresponding object segmentation masks. GLaMM not only grounds objects appearing in the con-versations but is flexible enough to accept both textual and optional visual prompts (region of interest) as input. This empowers users to interact with the model at various levels of granularity, both in textual and visual domains. Due to the lack of standard benchmarks for the novel setting of visually Grounded Conversation Generation (GCG), we in-troduce a comprehensive evaluation protocol with our curated grounded conversations. Our proposed GCG task requires densely grounded concepts in natural scenes at a large-scale. To this end, we propose a densely annotated Grounding-anything Dataset (GranD) using our proposed automated annotation pipeline that encompasses 7.5M unique concepts grounded in a total of 810M regions available with segmentation masks. Besides GCG, GLaMM also performs effectively on several downstream tasks, e.g., referring expression segmentation, image and region-level captioning and vision-language conversations.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6ae4705139494fcb6b790b6dd6c4225b40ee40f8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Hongxu Yin
2. Ruihan Yang
3. Peng Jin
4. Jiaxi Cui
5. Yang Fu
6. Munan Ning
7. Junwu Zhang
8. Bin Lin

Citations in the corpus

• A survey on multimodal large language models (2023, unknown): “In order to further inspect what roles LLMs exactly play in LLM-Aided Visual Reasoning systems, existing related works are divided into three types: • LLM as a Controller • LLM as a Decision Maker”
• Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, unknown): “Significant progress has been made in the field of general-domain image-text M-LLMs through the development of models such as GLaMM [71], Qwen-VL [72], SpatialVLM [73], InternVL [74], Osprey [75], Vary [76], ShareGPT4V [77], OtterHD [78], LION [79], SPHINX [80], BLIVA [81], SVIT [82], LLaVA [46],…”
• MoE-LLaVA: Mixture of Experts for Large Vision-Language Models (2024, background): “…LVLMs with fine-grained image understanding capabilities, such as region understanding (Chen et al., 2023c; Zhao et al., 2023b; Liu et al., 2023e), multi-region understanding (Wang et al., 2023c; Pi et al., 2023; Peng et al., 2023), and pixel-wise grounding (Rasheed et al., 2023; Lai et al., 2023).”
• Visual-RFT: Visual Reinforcement Fine-Tuning (2025, unknown): “Pioneered by LISA [13], there have been works done to enable large language models (LLMs) to output control tokens for other models (such as SAM [11]) or directly predict bounding box coordinates [31, 40] through supervised fine-tuning.”
• GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest (2023, unknown): “A multi-level feature pyramid (Lin et al., 2017) is constructed based on ViT-L/14.”
• SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, methodology): “In contrast, VisionLLM [27], GPT4RoI [28], [29], and Ferret [30, 31], along with GLaMM [32], use spatial boxes with ROI-aligned features to map region-level features into the LLM word embedding space.”
• VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, unknown): “Grounded caption data consist of textual descriptions of objects with corresponding bounding box annotations, primarily constructed from the Flickr-30K [79] and GranD [80] datasets.”