Kosmos-2: Grounding Multimodal Large Language Models to the World

By Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Furu Wei. 2023

1,249 citations recorded by the source corpus.

We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual world. Specifically, we represent refer expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where object descriptions are sequences of location tokens. Together with multimodal corpora, we construct large-scale data of grounded image-text pairs (called GrIT) to train the model. In addition to the existing capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning), Kosmos-2 integrates the grounding capability into downstream applications. We evaluate Kosmos-2 on a wide range of tasks, including (i) multimodal grounding, such as referring expression comprehension, and phrase grounding, (ii) multimodal referring, such as referring expression generation, (iii) perception-language tasks, and (iv) language understanding and generation. This work lays out the foundation for the development of Embodiment AI and sheds light on the big convergence of language, multimodal perception, action, and world modeling, which is a key step toward artificial general intelligence. Code and pretrained models are available at https://aka.ms/kosmos-2.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest (2023, neutral): “As shown in Table 3, we give the comparison of methods including InstructBLIP (Dai et al., 2023) with a pure language reference, Shikra (Chen et al., 2023b) with textual coordinates as reference, Kosmos-2 (Peng et al., 2023) with extra position tokens.”
SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model (2025, neutral): “Recent advances in Vision-Language-Action (VLA) models [7, 30, 5, 33] show a promising paradigm in building such generalist policy by fine-tuning the pre-trained Vision-Language Models (VLMs) [1, 55, 50, 37] on diverse robot data [13, 29, 18].”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, methodology): “Vision large language models (VLLMs) [3, 5, 19, 21, 28, 69, 87, 113, 147], which leverage LLMs, have also achieved significant breakthroughs, enabling sophisticated vision-language dialogues and interactions.”
SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, methodology): “Additionally, we include KOSMOS-2 [21], a VLM capable of taking bounding box inputs to reference objects, and RegionVILA (RegionGPT [16] with VILA [46] pre-training).”
LongNet: Scaling Transformers to 1, 000, 000, 000 Tokens (2023, methodology): “In the future, we will extend L ONG N ET to support more tasks, e.g., multimodal large language modeling [HDW + 23, PWD + 23], BEiT pretraining [BDPW22, PDB + 22, WBD + 23], and genomic data modeling.”
MoE-LLaVA: Mixture of Experts for Large Vision-Language Models (2024, background): “…LVLMs with fine-grained image understanding capabilities, such as region understanding (Chen et al., 2023c; Zhao et al., 2023b; Liu et al., 2023e), multi-region understanding (Wang et al., 2023c; Pi et al., 2023; Peng et al., 2023), and pixel-wise grounding (Rasheed et al., 2023; Lai et al., 2023).”
LLaVA-Prumerge: Adaptive Token Reduction for Efficient Large Multimodal Models (2024, neutral): “More recent works extend whole-image understanding into region-level understanding [6, 7, 36, 54], video understanding [26, 53] and 3D scene understanding [18].”
LISA: Reasoning Segmentation via Large Language Model (2023, background): “Kosmos-2 (Peng et al., 2023) constructs large-scale data of grounded image-text pairs, infusing grounding capabilities into LLMs.”
Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation (2024, background): “Recently, vision-and-language models [Liu et al. 2024; Zhu et al. 2023; Dai et al. 2024; Peng et al. 2023; Zhang et al. 2023; Ma et al. 2024] have achieved versatile visual understanding through visual instruction tuning [Liu et al. 2024; Zhu et al. 2023].”
Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities (2023, methodology): “A.3 Grounding For the GRIT (Peng et al., 2023) dataset, we found that there are many recursive grounding box labels in one caption.”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “…[211], Laion-ZH [203], Laion-EN [203], Laion-COCO [204], LLaVAR [305], InternVL-SA-1B-Caption [113], Captioning MMInstruct [155], GRIT-Caption [194], ShareGPT4V [29], LVIS-Instruct-4V [244], ShareCaptioner [29], OmniCorpus [133], ShareGPT4o [35] GQA [98], OKVQA [178], A-OKVQA [205], Visual7W…”
How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “To equip the model with high-resolution processing and OCR capabilities, both the vision encoder and the MLP were activated for training, utilizing a mix of image captioning [12, 19, 86, 89, 96] and OCR-specific datasets [29, 90].”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, background): “More recently, some approaches have been developed towards visual-interactive multimodal comprehension by precise referring instruction tuning (Zhao et al., 2023a; Peng et al., 2023; Chen et al., 2023a; Zhang et al., 2023f).”
SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension (2023, background): “With the impressive success of Large language models (LLM) [1, 5, 4], recent studies work on generative Multimodal Large Language Models (MLLMs) [6, 7, 8, 9, 10, 11, 12, 13, 14, 18, 19, 20, 21] to improve multimodal comprehension and generation through utilizing the strong generality of LLMs.”