3D-LLM: Injecting the 3D World into Large Language Models

By Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, Chuang Gan. Neural Information Processing Systems, 2023

599 citations recorded by the source corpus.

Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 1M 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi-view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on held-out evaluation dataset, ScanQA, SQA3D and 3DMV-VQA, outperform state-of-the-art baselines. In particular, experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin ( e.g. , the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs. Project Page: : https://vis-www.cs.umass.edu/3dllm/ .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/7637ed79d30d0139901175ae4abedd822c217ab4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (2024, methodology): “For the 3D data, we widely gather the training set from nuScenes QA [6], ALFRED [48], ScanQA [3], and 3D-LLM [16], covering both outdoor and indoor scenarios.”
PointLLM: Empowering Large Language Models to Understand Point Clouds (2023, background): “Concurrent with our work, 3D-LLM [16] seeks to enable large language models to comprehend the 3D world.”
MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2024, background): “They extend LLMs to tackle a diverse range of tasks and domains, including the mainstream 2D images [35, 15, 1, 31] and other modalities, such as 3D point clouds [23, 60, 27], audio [24, 52], and video [65, 6].”
LLaVA-Prumerge: Adaptive Token Reduction for Efficient Large Multimodal Models (2024, unknown): “Vision Transformers (ViTs) [12] are the most widely used vision encoder for LMMs, in which the input image is converted into a sequence of representative tokens by the ViT, and then fed into an LLM for understanding [18, 29, 52, 55].”
SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, background): “Recently, there has been a significant push to obtain spatial reasoning capabilities using LLMs. Initiatives [17, 18] have focused on reconstructing scenes from multi-view images, such as point clouds or neural fields, and enhancing these representations with dense semantic features.”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, background): “…content creation (Poole et al., 2023; Qi et al., 2023b; Liu et al., 2023d), representation learning (Dong et al., 2023; Qi et al., 2023a; Zhang et al., 2023a;f), scene comprehension (Zhang et al., 2023b; Hong et al., 2023), and embodied artificial inteligence (Ichter et al., 2022; Qi et al., 2024).”
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2023, background): “…the 2D representations produced by image and language foundation models to the 3D world and show impressive results on open-vocabulary tasks, including language-guided object grounding [17], [18], [24], [26], [36], 3D reasoning [37], [38], robot manipulation [39]–[41] and navigation [42], [43].”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “NeRF can be extracted from videos and contains rich 3D information for robot learning, as exemplified by F3RM [56] and 3D-LLM [57].”
3D-VLA: A 3D Vision-Language-Action Generative World Model (2024, unknown): “In the first stage, we develop the 3D-VLA base model following the methodology of 3D-LLM (Hong et al., 2023).”
Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “For example in the perception domain, Large Vision-Language Models (VLMs) have been found to provide cross-modal understanding by learning associations between visual and textual data, aiding tasks such as zero-shot image classiﬁcation, zero-shot object detection [10], and 3D classiﬁcation [11].”
SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model (2025, unknown): “Some researches [73, 10, 19, 24, 53, 25, 69] have focused on extending the generalist ability of LLMs and VLMs from language-vision towards the 3D world.”
OpenEQA: Embodied Question Answering in the Era of Foundation Models (2024, unknown): “(1) How closely aligned is the LLM-Match metric with human evaluators?”
LRM: Large Reconstruction Model for Single Image to 3D (2023, unknown): “There are also recent works in connecting 3D and large language models, such as 3D-LLM (Hong et al., 2023) and LLM-Grounder (Yang et al., 2023).”
LLaVA-OneVision: Easy Visual Task Transfer (2024, unknown): “We also utilize several multi-view benchmarks for evaluation, which depict 3D environments with multiple viewpoints, including 3D Dialogue (3D-Chat) and Task Decomposition (3D-TD) from 3D-LLM [38], ScanQA [5], ALFRED [122], and nuScenes VQA [9].”
A survey on multimodal large language models (2023, unknown): “In order to further inspect what roles LLMs exactly play in LLM-Aided Visual Reasoning systems, existing related works are divided into three types: • LLM as a Controller • LLM as a Decision Maker”
An Embodied Generalist Agent in 3D World (2023, unknown): “Concurrent works (Yin et al., 2023; Hong et al., 2023; Wang et al., 2023e; Xu et al., 2023) extend to 3D vision tasks, but these models either lack the acting capability or unified efficient architecture.”
EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain (2024, background): “D point clouds [31], [32], [33] are being integrated and aligned.”
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models (2023, background): “…parsing (Chen et al., 2023b; Peng et al., 2023b), in-context learning (Li et al., 2023a;b), arbitrary image resolutions (Bavishi et al., 2023), text-to-image generation (Wen et al., 2023; Dong et al., 2023), and 3D question answering (Xu et al., 2023; Guo et al., 2023; Hong et al., 2023).”