ImageBind One Embedding Space to Bind Them All

By Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, Ishan Misra. 2023

1,661 citations recorded by the source corpus.

We present ImageBind, an approach to learn a joint embedding across six different modalities - images, text, audio, depth, thermal, and IMU data. We show that all combinations of paired data are not necessary to train such a joint embedding, and only image-paired data is sufficient to bind the modalities together. ImageBind can leverage recent large scale vision-language models, and extends their zero-shot capabilities to new modalities just by using their natural pairing with images. It enables novel emergent applications ‘out-of-the-box’ including cross-modal retrieval, composing modalities with arithmetic, cross-modal detection and generation. The emergent capabilities improve with the strength of the image encoder and we set a new state-of-the-art on emergent zero-shot recognition tasks across modalities, outperforming specialist supervised models. Finally, we show strong few-shot recognition results outperforming prior work, and that ImageBind serves as a new way to evaluate vision models for visual and non-visual tasks.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Language Is Not All You Need: Aligning Perception with Language Models (2023, background): “In recent years, vision-language learning and representation models has garnered significant attention [2, 3, 4, 34, 50, 51, 52, 53, 54].”
Instruction Tuning for Large Language Models: A Survey (2023, neutral): “Scalability and Depth of Instruction Fine-Tuning.”
Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving (2023, background): “Another noteworthy advancement [33] involves the fusion of information from six distinct modalities: text, image/video, audio, depth, thermal, and inertial measurements.”
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models (2023, methodology): “For more modality input, ImageBind-LLM (Han et al., 2023) and PandaGPT (Su et al., 2023) further incorporate audio and video conditions guided by ImageBind (Girdhar et al., 2023).”
AnomalyGPT: Detecting Industrial Anomalies using Large Vision-Language Models (2023, methodology): “PandaGPT [25] establishes a connection between Image-Bind [8] and the Vicuna [3] model via a linear layer, allowing for multi-modal input.”
GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest (2023, neutral): “Wit-nessing the power of LLM, the field of multimodal models (Yang et al., 2023c; Huang et al., 2023; Girdhar et al., 2023; Driess et al., 2023) is developing a new technology direction to leverage LLM as the universal interface to build general-purpose models, where the feature space of a specific…”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, methodology): “Video-LLaMA [106] further leverage pre-trained models ImageBind [28] and LLaMA [82] bootstraping cross-modal training in video following BLIP-2.”
A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “Many have made significant progress [139], [147], but still lacks generalization capability on par with LLMs in NLP. Attaining [231] and LanguageBind [232], align different modalities to the image or language embedding space, respectively.”
PointLLM: Empowering Large Language Models to Understand Point Clouds (2023, methodology): “Such advancements may open up possibilities for more accessible 3D design across various applications and will bring 3D perception and creation closer to a wider range of users and industries.”
Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities (2023, background): “In addition to vision-language learning, ImageBind (Girdhar et al., 2023) and ONE-PEACE (Wang et al.”
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (2023, positive): “The pretrained ImageBind [21] is adopted as the multimodal encoder, which has an attribute of modality alignment.”
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (2024, methodology): “To do so, we use ImageBind [25] to extract embeddings of video frames and merge the adjacent clips if the frame embeddings from two clips are similar.”
ImageBind-LLM: Multi-modality Instruction Tuning (2023, methodology): “In this paper, we introduce a multi-modality instruction-following model, ImageBind-LLM, which efficiently fine-tunes LLaMA, guided by the joint embedding space in the pre-trained ImageBind [12].”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “A notable example is PandaGPT [50], which uniquely manages to interpret 6 different modalities at the same time, achieved through the integration of a multimodal encoder known as ImageBind [51].”
Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, background): “Recently, with the development of multi-modal LLMs [98], [260], leveraging LLMs for modality alignment holds promise in this regard.”
VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, methodology): “Models such as PandaGPT (Su et al., 2023) and Video-LLaMA (Zhang et al., 2023) utilize pre-trained systems like ImageBind (Girdhar et al., 2023) for a universal modality alignment, while MacawLLM (Lyu et al., 2023) optimizes separate branches for visual and audio data using diverse instructional…”
Video-LLaVA: Learning United Visual Representation by Alignment Before Projection (2023, background): “They bind other modalities to the image space through ImageBind’s [11] modality encoder.”
A Survey on Hallucination in Large Vision-Language Models (2024, background): “Enriching Modalities Integrating multi-modal signals not only expands the capabilities of models but also notably enhances their performance on single-modal tasks [Girdhar et al. , 2023].”