Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

By Ajay Jain, Matthew Tancik, Pieter Abbeel. IEEE International Conference on Computer Vision, 2021

640 citations recorded by the source corpus.

We present DietNeRF, a 3D neural scene representation estimated from a few images. Neural Radiance Fields (NeRF) learn a continuous volumetric representation of a scene through multi-view consistency, and can be rendered from novel viewpoints by ray casting. While NeRF has an impressive ability to reconstruct geometry and fine details given many images, up to 100 for challenging 360° scenes, it often finds a degenerate solution to its image reconstruction objective when only a few input views are available. To improve few-shot quality, we propose DietNeRF. We introduce an auxiliary semantic consistency loss that encourages realistic renderings at novel poses. DietNeRF is trained on individual scenes to (1) correctly render given input views from the same pose, and (2) match high-level semantic attributes across different, random poses. Our semantic loss allows us to supervise DietNeRF from arbitrary poses. We extract these semantics using a pre-trained visual encoder such as CLIP, a Vision Transformer trained on hundreds of millions of diverse single-view, 2D photographs mined from the web with natural language supervision. In experiments, DietNeRF improves the perceptual quality of few-shot view synthesis when learned from scratch, can render novel views with as few as one observed image when pre-trained on a multi-view dataset, and produces plausible completions of completely unobserved regions. Our project website is available at https://www.ajayj.com/dietnerf.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/81918488b569df3a43fd998d7b698fa9f6d1ff1b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

ReconFusion: 3D Reconstruction with Diffusion Priors (2023, positive): “Few-view NeRF Minimizing NeRF’s need for dense capture is crucial for democratizing 3D capture, and has motivated many works [10, 16, 22, 27, 35, 41, 47, 49, 50, 58, 62, 64, 65, 69].”
MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model (2022, background): “Recently, another generative model has attracted much attention with the considerable success achieved by NeRF [42], [43] in rendering realistic images.”
Zero-Shot Text-Guided Object Generation with Dream Fields (2021, methodology): “DietNeRF [24] proposed a related semantic consistency regularizer for NeRF based on the idea that “a bulldozer is a bulldozer from any perspective”.”
FreeNeRF: Improving Few-Shot Neural Rendering with Free Frequency Regularization (2023, methodology): “For instance, external models can be used to acquire normalization-ﬂow regularization [22], perceptual regularization [38], depth supervision [6, 29, 34], and cross-view semantic consistency [11].”
MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction (2022, background): “Previous works address this problem by incorporating priors on the structure of the scene into the optimization process, e.g., depth smoothness [46], surface smoothness [49,91], semantic similarity [30], or Manhattan world assumptions [21].”
DenseCLIP: Extract Free Dense Labels from CLIP (2021, background): “Various downstream vision tasks, e.g ., text-driven image manipulation [42], image captioning [25], view synthesis [30], and object detection [19], have attempted to exploit such features for improved generality and robustness.”
CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields (2021, methodology): “Furthermore, DietNeRF [12] designs a CLIP semantic consistency loss to improve few-shot NeRF and presents impressive results, and GRAF [37] ﬁrst adopts shape and appearance codes to conditionally synthesize NeRF, which inspires our adversarial training.”
DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data (2023, background): “CLIP [64] learns to map images and text captions into a shared embedding space, proving useful for many (often zero-shot) tasks [40, 65, 86].”
NeRF-Editing: Geometry Editing of Neural Radiance Fields (2022, background): “cn) generalization to different scenes [78], relighting [4,60], applying to dynamic scenes [48], and reducing the number of inputs [29].”
Splatter Image: Ultra-Fast Single-View 3D Reconstruction (2023, background): “When more than one view is available at the input, one can learn to interpolate between available views in the 3D space to estimate the scene geometry [5, 21], learn a view interpolation function [41] or optimize a 3D representation of a scene using semantic priors [11].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, positive): “DietNeRF [60] (June 2021) introduced the semantic consistency loss L sc based on image features extracted from Clip-ViT [100], in addition to the standard photometric loss. where φ performs the Clip-ViT feature extraction on training image I and rendered image ˆ I .”
Decomposing NeRF for Editing via Feature Field Distillation (2022, methodology): “Pre-trained vision models have been used for regularizing NeRF via flows [62], multi-view consistency [35], perceptual loss [110], or depth estimation [100, 77].”
Extract Free Dense Labels from CLIP (2021, background): “age manipulation [42], image captioning [25], view synthesis [30], and object detection [19].”
FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting (2023, methodology): “DietNeRF [23] adds supervision on high-dimensional semantic space, the CLIP embedding space [36], to constraint the rendered unseen views.”
DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization (2024, methodology): “We follow the data split used in [16, 52] for the Blender dataset [11].”
Depth-supervised NeRF: Fewer Views and Faster Training for Free (2021, background): “Another approach is to leverage priors recovered from a different task like semantic consistency [7] or depth prediction [30].”