SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models

By Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen and 6 more. 2023

306 citations recorded by the source corpus.

We present SPHINX, a versatile multi-modal large language model (MLLM) with a joint mixing of model weights, tuning tasks, and visual embeddings. First, for stronger vision-language alignment, we unfreeze the large language model (LLM) during pre-training, and introduce a weight mix strategy between LLMs trained by real-world and synthetic data. By directly integrating the weights from two domains, the mixed LLM can efficiently incorporate diverse semantics with favorable robustness. Then, to enable multi-purpose capabilities, we mix a variety of tasks for joint visual instruction tuning, and design task-specific instructions to avoid inter-task conflict. In addition to the basic visual question answering, we include more challenging tasks such as region-level understanding, caption grounding, document layout detection, and human pose estimation, contributing to mutual enhancement over different scenarios. Additionally, we propose to extract comprehensive visual embeddings from various network architectures, pre-training paradigms, and information granularity, providing language models with more robust image representations. Based on our proposed joint mixing, SPHINX exhibits superior multi-modal understanding capabilities on a wide range of applications. On top of this, we further propose an efficient strategy aiming to better capture fine-grained appearances of high-resolution images. With a mixing of different scales and high-resolution sub-images, SPHINX attains exceptional visual parsing and reasoning performance on existing evaluation benchmarks. We hope our work may cast a light on the exploration of joint mixing in future MLLM research. Code is released at https://github.com/Alpha-VLLM/LLaMA2-Accessory.

Subfields: Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

1. Bo Li
2. Chang Zhou
3. Peng Gao
4. Hao Shao
5. Renrui Zhang
6. Yuanhan Zhang
7. Yu Liu
8. Shuai Bai

In Multimodal Learning

1. Peng Gao
2. Hongsheng Li
3. Renrui Zhang
4. Pan Lu
5. Bo Li
6. Chang Zhou
7. Bo Zhang
8. Hongxu Yin

Citations in the corpus

• Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (2024, methodology): “This enthusiasm has led to the emergence of multi-modal large language models (MLLM), such as LLaVA [33, 34], SPHINX [14, 31], and Qwen-VL [3].”
• DeepSeek-VL: Towards Real-World Vision-Language Understanding (2024, background): “To address these limitations, recent researches (Lin et al., 2023b; Tong et al., 2024; Wei et al., 2023) have advocated for the integration of additional vision-only self-supervised encoders, to enhance the visual grounding capabilities of multi-modality models.”
• EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain (2024, methodology): “5 [6], Sphinx [77], and Qwen-VL-Chat [97] incorrectly categorize the storage tank as a stadium or residential area.”
• VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, background): “Models capable of image recognition (Bai et al., 2023; Dong et al., 2024; Chen et al., 2023d; Liu et al., 2024c; Chen et al., 2023b; Lin et al., 2023c; Young et al., 2024) and photorealistic image generation (Esser et al., 2024; Saharia et al., 2022; Ramesh et al., 2021) have approached near-human…”
• SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, background): “KOSMOS-2 [21], Shikra [22], MiniGPT-2 [23], CogVLM [24], SPHINX [25], and LLaVA [26] have enabled MLLMs to achieve region-based image understanding.”
• Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, unknown): “…has been made in the field of general-domain image-text M-LLMs through the development of models such as GLaMM [71], Qwen-VL [72], SpatialVLM [73], InternVL [74], Osprey [75], Vary [76], ShareGPT4V [77], OtterHD [78], LION [79], SPHINX [80], BLIVA [81], SVIT [82], LLaVA [46], and CoVLM [83].”
• SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics (2025, unknown): “…to other modalities, sparking interest towards multi-modal vision-language (VLMs) (Alayrac et al., 2022; Chen et al., 2023; Huang et al., 2023; Liu et al., 2023b; Chen et al., 2024; Shukor et al., 2023b) and audio-language models (ALMs) (Défossez et al., 2024; Das et al., 2024; Borsos et al.,…”
• MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2024, background): “The subsequent mPLUG-Owl [61, 62], Qwen-VL [3], InternLM-XComposer [16], and SPHINX [36, 21] further push the frontier of MLLMs in understanding and generalizing across visual contexts.”
• How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, unknown): “Currently, models like CLIP-ViT [87] and SigLIP [131] are prevalently utilized; however, many studies have been conducted to find the most suitable vision encoders for MLLMs [53, 67, 72, [58, 60], combining a pre-trained InternViT-6B [20] with InternLM2-20B [13] through a MLP projector.”
• CogVLM: Visual Expert for Pretrained Language Models (2023, unknown): “…et al., 2015], OKVQA [Marino et al., 2019], TextVQA [Singh et al., 2019], OCRVQA [Mishra et al., 2019], ScienceQA [Lu et al., 2022], as well as datasets formatted as multi-turn dialogues such as LLaVA-Instruct [Liu et al., 2023c], LRV-Instruction [Liu et al., 2023a], LLaVAR [Zhang et al., 2023b].”
• SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation (2024, unknown): “In recent years, Multimodal Large Language Models (MLLMs) [1, 2, 3, 4, 5, 6, 7, 8] have demonstrated exceptional capabilities in comprehending multimodal data through leveraging the strong generality of LLMs [9, 10, 11].”
• What matters when building vision-language models? (2024, unknown): “This results in a high number of visual tokens for each image ranging from 576 for DeepSeek-VL (Lu et al., 2024) to 2890 for SPHINX-2k (Lin et al., 2023).”
• TinyVLA: Toward Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation (2024, unknown): “TinyVLA encompasses several crucial designs: 1) We adopt a pre-trained VLM as the initialization of a policy network; 2) During training the robot data, we freeze the pre-trained parts and utilize the parameter-efficient fine-tuning technique LoRA [29], where the trainable parameters account for only…”
• LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (2024, background): “However, most open-source LMMs [11,24,34,36] have primarily focused on pushing the performance limit of the single-image scenario, the more complex multi-image scenarios remain largely less explored.”