LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment

By Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, Hongfa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zong-Rui Li and 4 more. International Conference on Learning Representations, 2023

451 citations recorded by the source corpus.

The video-language (VL) pretraining has achieved remarkable improvement in multiple downstream tasks. However, the current VL pretraining framework is hard to extend to multiple modalities (N modalities, N>=3) beyond vision and language. We thus propose LanguageBind, taking the language as the bind across different modalities because the language modality is well-explored and contains rich semantics. Specifically, we freeze the language encoder acquired by VL pretraining, then train encoders for other modalities with contrastive learning. As a result, all modalities are mapped to a shared feature space, implementing multi-modal semantic alignment. While LanguageBind ensures that we can extend VL modalities to N modalities, we also need a high-quality dataset with alignment data pairs centered on language. We thus propose VIDAL-10M with Video, Infrared, Depth, Audio and their corresponding Language, naming as VIDAL-10M. In our VIDAL-10M, all videos are from short video platforms with complete semantics rather than truncated segments from long videos, and all the video, depth, infrared, and audio modalities are aligned to their textual descriptions. After pretraining on VIDAL-10M, we outperform ImageBind by 5.8% R@1 on the MSR-VTT dataset with only 15% of the parameters in the zero-shot video-text retrieval task. Beyond this, our LanguageBind has greatly improved in the zero-shot video, audio, depth, and infrared understanding tasks. For instance, LanguageBind surpassing InterVideo by 1.9% on MSR-VTT, 8.8% on MSVD, 6.3% on DiDeMo, and 4.4% on ActivityNet. On the LLVIP and NYU-D datasets, LanguageBind outperforms ImageBind with 23.8% and 11.1% top-1 accuracy. Code address: https://github.com/PKU-YuanGroup/LanguageBind.

Subfields: Cancer Research, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8eb99f1ed884356871ddbcf1377b82359071906a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Peng Jin
2. Munan Ning
3. Bin Lin
4. Bin Zhu
5. Li Yuan
6. Caiwan Zhang
7. Yang Ye
8. Xiaochun Cao

Citations in the corpus

• MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding (2024, unknown): “Subsequently, we employ a robust video feature extraction tool, LanguageBind [54] to extract clip embeddings F I ∈ R N × d , where d represents the dimension of each clip embedding.”
• Video-LLaVA: Learning United Visual Representation by Alignment Before Projection (2023, methodology): “2, Video-LLaVA consists of LanguageBind encoders f V M [54] to extract features from the raw visual signal ( e.g . images or videos), a large language model f L such as Vicuna, visual projection layers f P and a word embedding layer f T .”
• Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (2023, background): “Visual information serves to augment and enhance our comprehension of the world [5, 23–26, 29, 59].”
• LLaVA-Video: Video Instruction Tuning With Synthetic Data (2024, unknown): “…2017; Xue et al., 2022; Goyal et al., 2017; Caba Heilbron et al., 2015; Kay et al., 2017; Sigurdsson et al., 2016; Wang et al., 2023; Shang et al., 2019; Grauman et al., 2022; Zhu et al., 2023a), covering a wide range of video domains, such as activities, cooking, TV shows, and egocentric views.”
• A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “Many have made significant progress [139], [147], but still lacks generalization capability on par with LLMs in NLP. Attaining [231] and LanguageBind [232], align different modalities to the image or language embedding space, respectively.”
• Video Instruction Tuning With Synthetic Data (2024, unknown): “…2017; Xue et al., 2022; Goyal et al., 2017; Caba Heilbron et al., 2015; Kay et al., 2017; Sigurdsson et al., 2016; Wang et al., 2023; Shang et al., 2019; Grauman et al., 2022; Zhu et al., 2023a), covering a wide range of video domains, such as activities, cooking, TV shows, and egocentric views.”
• CFVBench: A Comprehensive Video Benchmark for Fine-grained Multimodal Retrieval-Augmented Generation (2025, unknown): “To thoroughly assess existing video-based MRAG methods’ performance, we separately evaluate both stages of MRAG on the proposed CFVBench.”