ConceptFusion: Open-set Multimodal 3D Mapping

By Krishna Murthy Jatavallabhula, Ali Kuwajerwala, Qiao Gu, Mohd. Omama, Tao Chen, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Varma Keetha, Ayush Kumar Tewari and 6 more. Robotics: Science and Systems Conference, 2023

408 citations recorded by the source corpus.

Building 3D maps of the environment is central to robot navigation, planning, and interaction with objects in a scene. Most existing approaches that integrate semantic concepts with 3D maps largely remain confined to the closed-set setting: they can only reason about a finite set of concepts, pre-defined at training time. Further, these maps can only be queried using class labels, or in recent work, using text prompts. We address both these issues with ConceptFusion, a scene representation that is (1) fundamentally open-set, enabling reasoning beyond a closed set of concepts and (ii) inherently multimodal, enabling a diverse range of possible queries to the 3D map, from language, to images, to audio, to 3D geometry, all working in concert. ConceptFusion leverages the open-set capabilities of today's foundation models pre-trained on internet-scale data to reason about concepts across modalities such as natural language, images, and audio. We demonstrate that pixel-aligned open-set features can be fused into 3D maps via traditional SLAM and multi-view fusion approaches. This enables effective zero-shot spatial reasoning, not needing any additional training or finetuning, and retains long-tailed concepts better than supervised approaches, outperforming them by more than 40% margin on 3D IoU. We extensively evaluate ConceptFusion on a number of real-world datasets, simulated home environments, a real-world tabletop manipulation task, and an autonomous driving platform. We showcase new avenues for blending foundation models with 3D open-set multimodal mapping. For more information, visit our project page https://concept-fusion.github.io or watch our 5-minute explainer video https://www.youtube.com/watch?v=rkXgws8fiDs

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/5e2bceb56f116e98baf7e418208057bc0e1c1861

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “Real : SayCan setup LEO [159] ConvNeXt [160], PointNet++ [161] Vicuna-7B Concat MLP Yes Sim-Mani : CLIPort task subset; Sim-Navi : Habitat-web 3D-LLM [57] CLIP, 3D-CLR [162], ConceptFusion [163] Flamingo, BLIP-2”
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2023, methodology): “…the 2D representations produced by image and language foundation models to the 3D world and show impressive results on open-vocabulary tasks, including language-guided object grounding [17], [18], [24], [26], [36], 3D reasoning [37], [38], robot manipulation [39]–[41] and navigation [42], [43].”
OpenMask3D: Open-Vocabulary 3D Instance Segmentation (2023, methodology): “Recent success of 2D open-vocabulary segmentation models such as LSeg [39], OpenSeg [18], and OV-Seg[41] has motivated researchers in the field of 3D scene understanding to explore the open vocabulary setting [6, 10, 17, 21, 27, 29, 32, 34, 45, 49, 56, 57].”
LangSplat: 3D Language Gaussian Splatting (2023, background): “The field of open-ended language queries in 3D has attracted increasing attention due to its various applications such as robotic navigation [14] and manipulation [35], 3D semantic understanding [9, 49] and editing [20], autonomous driving [16], and augmented/virtual reality [23].”
LERF: Language Embedded Radiance Fields (2023, methodology): “Concurrent work ConceptFusion [19] fuses CLIP features more densely in RGBD pointclouds, using Mask2Former [9] to predict regions of interest, meaning it can lose objects which are out of distribution to Mask2Former’s training set.”