Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic

By Ke Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, Rui Zhao. 2023

964 citations recorded by the source corpus.

In human conversations, individuals can indicate relevant regions within a scene while addressing others. In turn, the other person can then respond by referring to specific regions if necessary. This natural referential ability in dialogue remains absent in current Multimodal Large Language Models (MLLMs). To fill this gap, this paper proposes an MLLM called Shikra, which can handle spatial coordinate inputs and outputs in natural language. Its architecture consists of a vision encoder, an alignment layer, and a LLM. It is designed to be straightforward and simple, without the need for extra vocabularies, position encoder, pre-/post-detection modules, or external plug-in models. All inputs and outputs are in natural language form. Referential dialogue is a superset of various vision-language (VL) tasks. Shikra can naturally handle location-related tasks like REC and PointQA, as well as conventional VL tasks such as Image Captioning and VQA. Experimental results showcase Shikra's promising performance. Furthermore, it enables numerous exciting applications, like providing mentioned objects' coordinates in chains of thoughts and comparing user-pointed regions similarities. Our code, model and dataset are accessed at https://github.com/shikras/shikra.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

Improved Baselines with Visual Instruction Tuning (2023, methodology): “We also incorporate ShareGPT [45] data and scale up the LLM to 13B as in [3, 8, 38].”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, background): “Vision large language models (VLLMs) [3, 5, 19, 21, 28, 69, 87, 113, 147], which leverage LLMs, have also achieved significant breakthroughs, enabling sophisticated vision-language dialogues and interactions.”
SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, background): “KOSMOS-2 [21], Shikra [22], MiniGPT-2 [23], CogVLM [24], SPHINX [25], and LLaVA [26] have enabled MLLMs to achieve region-based image understanding.”
MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning (2023, methodology): “Our results compare favorably to baselines including MiniGPT-4 [59], Shikra [7], LLaVA [26], and InstructBLIP [10] across all the VQA tasks.”
SVIT: Scaling up Visual Instruction Tuning (2023, neutral): “5 to state-of-the-art MLLMs: BLIP-2 (Li et al., 2023c), InstructBLIP (Dai et al., 2023), Shikra (Chen et al., 2023), IDEFICS (IDEFICS, 2023), Qwen-VL(-Chat) (Bai et al., 2023), mPLUG-Owl2 (Ye et al., 2023) and LLaVA-v1.”
MMBench: Is Your Multi-modal Model an All-around Player? (2023, neutral): “For each sample, GPT score is an integer n ∈ [1, 5], indicating the similarity between answer and groundtruth.”
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, neutral): “Unlike previous works (Zhu et al., 2023a; Liu et al., 2023b;a) that utilize instruction-tuned LLMs such as Vicuna (Chiang et al., 2023), SPHINX-X is directly trained on top of the basic pre-trained LLM, i.e., LLaMA2 (Touvron et al., 2023b).”
LMDrive: Closed-Loop End-to-End Driving with Large Language Models (2023, background): “In the visual language community [5, 29, 58], the most common way to align vision and language could be us-ing pre-trained CLIP models [35] to encode image features.”
Generative Multimodal Models are In-Context Learners (2023, background): “A school of successive efforts [69, 76, 84, 90, 91] further improves visual instruction tuning with better overall training pipelines [6, 40], grounding annotations [14, 15, 85, 92], and extra tasks [6].”
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models (2023, methodology): “Low-resolution constraints of MLLMs. State-of-the-art open-source MLLMs (Li et al., 2023d; Liu et al., 2023d; Gao et al., 2023b; Chen et al., 2023b; Peng et al., 2023b; Chen et al., 2023a) works adopt frozen image encoders during all training stages, in order to preserve the pre-trained visual…”
MoE-LLaVA: Mixture of Experts for Large Vision-Language Models (2024, methodology): “…recent works have focused on improving performance through methods such as expanding the instruction-tuning dataset (Liu et al., 2023a;c; Zhang et al., 2023c; Zhao et al., 2023a; Chen et al., 2023d), optimizing training strategies (Bai et al., 2023b; Chen et al., 2023b), increasing…”
Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models (2023, methodology): “For visual modality, GPT4 (OpenAI, 2023), Flamingo (Alayrac et al., 2022), Kosmos (Peng et al., 2023), BLIP (Li et al., 2022), Shikra (Chen et al., 2023), Emu (Sun et al., 2023) and Qwen-VL (Bai et al., 2023) have proposed different integration method to enable image understanding or generation…”
Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (2023, background): “Existing large-scale multimodal models [4, 9–11, 17, 18, 31, 61, 74] broadly categorized into two classes.”
mPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration (2023, background): “∗ Equal contribution † Corresponding author 1 https://openai.com/research/gpt-4v-system-card benchmarks using a single generic model .”
GPT-4V(ision) is a Generalist Web Agent, if Grounded (2024, background): “We are strongly against any potentially harmful use of the data or technology by any party.”
MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (2023, methodology): “, 2023b) dataset generated by GPT-4 (OpenAI, 2023); (5) Shikra (Chen et al., 2023), a VLM that can handle spatial coordinate inputs and outputs in natural language without the need for extra vocabularies or external plugin models.”
GLaMM: Pixel Grounding Large Multimodal Model (2023, methodology): “A few recent works have explored grounded text response generation [5, 21, 35, 59] but do not provide detailed pixel-level groundings.”
Evaluating Object Hallucination in Large Vision-Language Models (2023, background): “Large language models (LLMs) (Zhao et al., 2023) have shown remarkable abilities to solve various complex tasks by following human instructions in a zero-shot manner.”