LERF: Language Embedded Radiance Fields

By Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, Matthew Tancik. IEEE International Conference on Computer Vision, 2023

691 citations recorded by the source corpus.

Humans describe the physical world using natural language to refer to specific 3D locations based on a vast range of properties: visual appearance, semantics, abstract associations, or actionable affordances. In this work we propose Language Embedded Radiance Fields (LERFs), a method for grounding language embeddings from off-the-shelf models like CLIP into NeRF, which enable these types of open-ended language queries in 3D. LERF learns a dense, multi-scale language field inside NeRF by volume rendering CLIP embeddings along training rays, supervising these embeddings across training views to provide multi-view consistency and smooth the underlying language field. After optimization, LERF can extract 3D relevancy maps for a broad range of language prompts interactively in real-time, which has potential use cases in robotics, understanding vision-language models, and interacting with 3D scenes. LERF enables pixel-aligned, zero-shot queries on the distilled 3D CLIP embeddings without relying on region proposals or masks, supporting long-tail open-vocabulary queries hierarchically across the volume. See the project website at: https://lerf.io.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/611e2100a3f8ed02b2583d4e53fd27a12d223b4c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

OpenMask3D: Open-Vocabulary 3D Instance Segmentation (2023, methodology): “On the other hand, approaches such as LERF [35] and DFF [37] leverage the interpolation capabilities of NeRFs [50] to extract a semantic field of the scene.”
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (2024, background): “Following the hypotheses in Kerr et al. (2023) and similar work, we believe that DINOv2 features provide features that capture low-level spatial properties of an image, augmenting the higher-level “semantic” properties captured by vision-language contrastive models.”
Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields (2023, methodology): “Beyond optimizing NeRF with estimated labels, Distilled Feature Fields [18], NeRF-SOS [9], LERF [16], and Neural Feature Fusion Fields [42] have delved into embedding pixel-aligned feature vectors from technologies such as LSeg or DINO [4] into NeRF frameworks.”
An Embodied Generalist Agent in 3D World (2023, neutral): “2 and Appendix B.6, and quantitative analysis on the impact of data refinement procedures in Appendix I.1.”
gsplat: An Open-Source Library for Gaussian Splatting (2024, neutral): “This is motivated by algorithms that combine learned feature maps with diﬀerentiable volume rendering (Kobayashi et al., 2022; Kerr et al., 2023).”
LangSplat: 3D Language Gaussian Splatting (2023, methodology): “With the number of scales s potentially reaching as high as 30 [18], this significantly diminishes the inference speed.”
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2023, background): “…the 2D representations produced by image and language foundation models to the 3D world and show impressive results on open-vocabulary tasks, including language-guided object grounding [17], [18], [24], [26], [36], 3D reasoning [37], [38], robot manipulation [39]–[41] and navigation [42], [43].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, neutral): “The model achieved good results on the CelebA dataset [115], the CARLA dataset [116], and the ShapeNet subclasses from SRN [119], but is outperformed by its backbone π -GAN for conditional image synthesis.”
LRM: Large Reconstruction Model for Single Image to 3D (2023, positive): “LERF (Kerr et al., 2023) learns a language field inside NeRF by rendering CLIP embeddings along training rays.”
Foundation models in robotics: Applications, challenges, and the future (2023, neutral): “…Anything [59], FastSAM [60], MobileSAM [61], Track Anything Model (TAM) [62] Open-Vocabulary 3D Scene Representation e.g. CLIP-NERF [63], LERF [64], DFF [65] Affordances e.g. Affordance Diffusion [66], VRB [67] Embodied AI e.g. Huang et al. [68], Statler [69], EmbodiedGPT [70], MineDojo [71],…”
Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, methodology): “3D scenes with existing open-vocabulary 3D segmentation method, such as LERF [14].”