pixelNeRF: Neural Radiance Fields from One or Few Images
By Alex Yu, Vickie Ye, Matthew Tancik, Angjoo Kanazawa. 2020
2,248 citations recorded by the source corpus.
We propose pixelNeRF, a learning framework that predicts a continuous neural scene representation conditioned on one or few input images. The existing approach for constructing neural radiance fields [27] involves optimizing the representation to every scene independently, requiring many calibrated views and significant compute time. We take a step towards resolving these shortcomings by introducing an architecture that conditions a NeRF on image inputs in a fully convolutional manner. This allows the network to be trained across multiple scenes to learn a scene prior, enabling it to perform novel view synthesis in a feed-forward manner from a sparse set of views (as few as one). Leveraging the volume rendering approach of NeRF, our model can be trained directly from images with no explicit 3D supervision. We conduct extensive experiments on ShapeNet benchmarks for single image novel view synthesis tasks with held-out objects as well as entire unseen categories. We further demonstrate the flexibility of pixelNeRF by demonstrating it on multi-object ShapeNet scenes and real scenes from the DTU dataset. In all cases, pixelNeRF outperforms current state-of-the-art baselines for novel view synthesis and single image 3D reconstruction. For the video and code, please visit the project website:https://alexyu.net/pixelnerf.
Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
- Andreas Geiger
- Matthias Nießner
- Michael Niemeyer
- Jonathan T. Barron
- Justin Johnson
- Andrea Tagliasacchi
- Songyou Peng
- Ben Mildenhall
In Computer Graphics and Computer-Aided Design
- Andreas Geiger
- Jonathan T. Barron
- Ben Mildenhall
- Pratul P. Srinivasan
- Matthias Nießner
- Michael Niemeyer
- Matthew Tancik
- Vincent Sitzmann
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Jitendra Malik
- Dacheng Tao
- Andrea Vedaldi
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Andrew Zisserman
In 3D Vision & Pose
- Andreas Geiger
- Matthias Nießner
- Michael Niemeyer
- Jonathan T. Barron
- Jitendra Malik
- Justin Johnson
- Angela Dai
- Andrea Vedaldi
In Meta & Continual Learning
- Zexiang Xu
- Andreas Geiger
- Andrea Tagliasacchi
- Deva Ramanan
- Jonathan T. Barron
- Andrea Vedaldi
- Marco Pavone
- Jiawei Yang
Citations in the corpus
- A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “Hopefully, with the further advancement of 3D learning-based approaches [126], [127] that aims to render 3D object meshes from a single or few images or even through scene generation approach [128], we will be able to scale up the collection process of large scale 3D datasets.”
- DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (2023, background): “The problem can also be formulated as single-view 3D reconstruction (Yu et al., 2021; Trevithick & Yang, 2021; Duggal & Pathak, 2022), but such reconstruction settings usually produce blurry results due to the lack of uncertainty modeling.”
- One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization (2023, methodology): “Among the methods, some [75, 61, 21, 89, 87, 37, 33, 70, 71] directly aggregate 2D features with MLPs or transformers, while others explicitly construct the 3D feature/cost volume [6, 28, 92, 40], and utilize the voxel feature for decoding density and color.”
- Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly- Throughs (2021, methodology): “PixelNeRF [36], IBRNet [31], and GRF [29] condition NeRF on predicted image features while Tancik et al.”
- MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo (2021, methodology): “In contrast, the concurrent works [54, 46] require retaining the input images for rendering.”
- CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, methodology): “We found that conditioning on input views through 3D self-attention layers removed the need for PixelNeRF [63] and CLIP image embeddings [64] used by the prior state-of-the-art model on few-view reconstruction, ReconFusion [7].”
- Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (2023, neutral): “Using a single reference image is insufficient to train a complete NeRF model without any priors [99, 45].”
- IBRNet: Learning Multi-View Image-Based Rendering (2021, background): “Concurrent work [58, 63] tries to address this issue and generalize NeRF (with a focus on very sparse input views).”
- Mixture of volumetric primitives for efficient neural rendering (2021, methodology): “2020], reducing data requirements [Trevithick and Yang 2020; Yu et al. 2020], and learning efficiency [Tancik et al.”
- Zero-1-to-3: Zero-shot One Image to 3D Object (2023, background): “On the other hand, locally conditioned models [46, 63, 54, 51, 52] aim”
- Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers (2023, neutral): “Instead of optimizing a 3D representation, alternative approaches utilize category-specific shape priors to regress 3D representations (e.g., point clouds [15, 70], voxels [19, 69], meshes [63, 67], and NeRF [77]) from single images.”
- Advances in Neural Rendering (2021, background): “…[CBLPM21] uses only a dozen spread-out views to generate novel views of a rigid scene with the visual accuracy comparable to the original NeRF [MST ∗ 20] after fine-tuning, while pixelNeRF [YYTK21] can infer volumetric models of rigid scenes unseen at training time just from a single image.”
- Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis (2021, methodology): “Even with 3 input views, NeRF [30] fails to learn accurate geometry or textures (reprinted from [52]).”
- Multi-view 3D Reconstruction with Transformers (2021, methodology): “Besides, our method is learned without requiring camera parameters, while coordinate-based implicit 3D representation methods are primarily for view synthesis and require camera parameters [14, 29, 24].”
- FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting (2023, background): “…compared to the previous SparseNeRF [50]. tions [11, 33, 47, 54] within the density field, additional supervision from 2D pre-trained models [23, 50], large-scale pre-training [8, 59], and frequency annealings [55] have been proposed and adopted to address the challenge of few-shot view synthesis.”
- FreeNeRF: Improving Few-Shot Neural Rendering with Free Frequency Regularization (2023, methodology): “Another thread of works [4, 5, 37] attempts to learn trans-ferable models by training on a large, curated dataset instead of using an external model.”