ImageBind-LLM: Multi-modality Instruction Tuning
By Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo and 7 more. 2023
181 citations recorded by the source corpus.
We present ImageBind-LLM, a multi-modality instruction tuning method of large language models (LLMs) via ImageBind. Existing works mainly focus on language and image instruction tuning, different from which, our ImageBind-LLM can respond to multi-modality conditions, including audio, 3D point clouds, video, and their embedding-space arithmetic by only image-text alignment training. During training, we adopt a learnable bind network to align the embedding space between LLaMA and ImageBind's image encoder. Then, the image features transformed by the bind network are added to word tokens of all layers in LLaMA, which progressively injects visual instructions via an attention-free and zero-initialized gating mechanism. Aided by the joint embedding of ImageBind, the simple image-text training enables our model to exhibit superior multi-modality instruction-following capabilities. During inference, the multi-modality inputs are fed into the corresponding ImageBind encoders, and processed by a proposed visual cache model for further cross-modal embedding enhancement. The training-free cache model retrieves from three million image features extracted by ImageBind, which effectively mitigates the training-inference modality discrepancy. Notably, with our approach, ImageBind-LLM can respond to instructions of diverse modalities and demonstrate significant language generation quality. Code is released at https://github.com/OpenGVLab/LLaMA-Adapter.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Alignment & RLHF, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Hongsheng Li
- Mohit Bansal
- Yu Qiao
- Peng Gao
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Hongsheng Li
- Xiaogang Wang
- Yu Qiao
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Jitendra Malik
In Multimodal Learning
Citations in the corpus
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “However, this full-model fine-tuning can be inefficient in both time and memory, limiting its transferability to downstream applications.”
- PointLLM: Empowering Large Language Models to Understand Point Clouds (2023, methodology): “To rigorously evaluate our model’s perceptual abilities and its generalization capabilities, we establish two benchmarks: Generative 3D Object Classification and 3D Object Caption-ing, assessed through three different methods, including”
- Video-LLaVA: Learning United Visual Representation by Alignment Before Projection (2023, methodology): “Expanding LLMs to additional visual modalities typically requires pre-alignment, as seen in LLaMA-Adapter [10, 53] and ImageBind-LLM [15].”
- A Survey on Hallucination in Large Vision-Language Models (2024, methodology): “Recent LVLMs research made strides in incorporating more modalities (e.g., videos) through methods including through cache retrieval [Han et al. , 2023] and pre-alignment [Lin et al. , 2023].”
- Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, methodology): “PandaGPT [170] and ImageBind-LLM [69] use the universal encoder ImageBind [56] to encode many kinds of input modalities, and ChatBridge [222] uses a similar universal encoder based on language.”
- MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2024, background): “They extend LLMs to tackle a diverse range of tasks and domains, including the mainstream 2D images [35, 15, 1, 31] and other modalities, such as 3D point clouds [23, 60, 27], audio [24, 52], and video [65, 6].”
- SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models (2023, unknown): “We test our model on recently proposed MLLM benchmarks to comprehensively evaluation of the model’s characteristic such as MME (Fu et al., 2023b), Seedbench (Li et al., 2023c), POPE (Li et al., 2023e), LLaVA-Bench (In-the-Wild) (Liu et al., 2023d), MM-Vet (Yu et al., 2023b), MathVista (Lu et al., 20”
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (2023, unknown): “2”
- LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (2024, background): “Traditionally, researchers have approached these challenges by training separate task-specific models for each application scenario, e.g., multi-image [1, 19, 27], video [7, 29, 66], and 3D [14, 15, 58].”
- A survey on multimodal large language models (2023, unknown): “To the best of our knowledge, this is the first survey on MLLM.”
- SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, unknown): “Many efforts also extend LLMs into more modalities, such as ImageBind-LLM (Han et al., 2023a), Point-LLM (Guo et al., 2023), and others (Zhu et al., 2023b; Zhang et al., 2022a; 2023b).”
- AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoning (2025, unknown): “Prominent open-source examples include ImageBind-LLM [Han et al. 2023], ChatBridge [Zhao et al. 2023], Macaw-LLM [Lyu et al. 2023a], VAST [Chen et al. 2023b], OneLLM [Han et al. 2024], VideoLLaMA2 [Cheng et al. 2024], and video-SALMONN [Sun et al. 2025b].”