Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following

By Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Ke Chen, Peng Gao, Xianzhi Li, Hongsheng Li and 1 more. 2023

235 citations recorded by the source corpus.

We introduce Point-Bind, a 3D multi-modality model aligning point clouds with 2D image, language, audio, and video. Guided by ImageBind, we construct a joint embedding space between 3D and multi-modalities, enabling many promising applications, e.g., any-to-3D generation, 3D embedding arithmetic, and 3D open-world understanding. On top of this, we further present Point-LLM, the first 3D large language model (LLM) following 3D multi-modal instructions. By parameter-efficient fine-tuning techniques, Point-LLM injects the semantics of Point-Bind into pre-trained LLMs, e.g., LLaMA, which requires no 3D instruction data, but exhibits superior 3D and multi-modal question-answering capacity. We hope our work may cast a light on the community for extending 3D point clouds to multi-modality applications. Code is available at https://github.com/ZiyuGuo99/Point-Bind_Point-LLM.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hongsheng Li
2. Yu Qiao
3. Peng Gao
4. Renrui Zhang
5. Jiaming Han
6. Dahua Lin
7. Ziyu Guo
8. Pan Lu

In Computer Vision and Pattern Recognition

1. Hongsheng Li
2. Yu Qiao
3. Xiaogang Wang
4. Luc Van Gool
5. Dacheng Tao
6. Dahua Lin
7. Chen Change Loy
8. Ming-Hsuan Yang

In Large Language Models

1. Bo Li
2. Yu Qiao
3. Peng Gao
4. Jiaming Han
5. Dahua Lin
6. Ziyi Lin
7. Renrui Zhang
8. Yuanhan Zhang

In Multimodal Learning

1. Peng Gao
2. Hongsheng Li
3. Yu Qiao
4. Renrui Zhang
5. Pan Lu
6. Bo Li
7. Jiaming Han
8. Siyuan Huang

In 3D Vision & Pose

1. Bo Li
2. Hongsheng Li
3. Yilun Chen
4. Yu Qiao
5. Siyuan Huang
6. Dahua Lin
7. Tai Wang
8. Xiangyu Yue

Citations in the corpus

• LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “The current working directory is /home/user/ echo “I am a chatbot“ > file.txt && cat file.txt && ls -l I am a chatbot -rw-r–r– 1 user user”
• EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain (2024, background): “D point clouds [31], [32], [33] are being integrated and aligned.”
• SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models (2023, methodology): “3) Employing advanced image encoders for extracting visual embeddings, e.g., the CLIP encoder (Radford et al., 2021) in LLaVA (Liu et al., 2023c) and the Q-Former (Li et al., 2023d) in MiniGPT-4 (Zhu et al., 2023).”
• A Survey on Multimodal Large Language Models for Autonomous Driving (2023, methodology): “Inspired by the success of pre-trained NLP models like BERT [34], T5 [142], and GPTs [14,140], researchers developed multi-modal Transformers that can process cross-modality inputs such as text, image, audio, pointcloud [45,51,59].”
• LLaVA-OneVision: Easy Visual Task Transfer (2024, unknown): “Specifically, most focus on pushing the performance limits in single-image scenarios [26, 83, 172, 73, 163, 35], only a few recent papers have begun to explore multi-image scenarios [70, 47].”
• ImageBind-LLM: Multi-modality Instruction Tuning (2023, methodology): “For instructions in 3D domains, we utilize the pre-trained 3D encoder in Point-Bind [13] to encode the input 3D point clouds.”
• LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (2024, background): “Traditionally, researchers have approached these challenges by training separate task-specific models for each application scenario, e.g., multi-image [1, 19, 27], video [7, 29, 66], and 3D [14, 15, 58].”
• SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, unknown): “Many efforts also extend LLMs into more modalities, such as ImageBind-LLM (Han et al., 2023a), Point-LLM (Guo et al., 2023), and others (Zhu et al., 2023b; Zhang et al., 2022a; 2023b).”
• MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2024, background): “They extend LLMs to tackle a diverse range of tasks and domains, including the mainstream 2D images [35, 15, 1, 31] and other modalities, such as 3D point clouds [23, 60, 27], audio [24, 52], and video [65, 6].”
• PointLLM: Empowering Large Language Models to Understand Point Clouds (2023, methodology): “This transformation can unlock applications in human-computer collaborative 3D generation, streamlining the process of 3D creation and reducing the dependency on specialized tools and expertise.”