iNeRF: Inverting Neural Radiance Fields for Pose Estimation

By Yen-Chen Lin, Peter R. Florence, Jonathan T. Barron, Alberto Rodriguez, Phillip Isola, Tsung-Yi Lin. IEEE/RJS International Conference on Intelligent RObots and Systems, 2020

586 citations recorded by the source corpus.

We present iNeRF, a framework that performs mesh-free pose estimation by "inverting" a Neural Radiance Field (NeRF). NeRFs have been shown to be remarkably effective for the task of view synthesis — synthesizing photorealistic novel views of real-world scenes or objects. In this work, we investigate whether we can apply analysis-by-synthesis via NeRF for mesh-free, RGB-only 6DoF pose estimation – given an image, find the translation and rotation of a camera relative to a 3D object or scene. Our method assumes that no object mesh models are available during either training or test time. Starting from an initial pose estimate, we use gradient descent to minimize the residual between pixels rendered from a NeRF and pixels in an observed image. In our experiments, we first study 1) how to sample rays during pose refinement for iNeRF to collect informative gradients and 2) how different batch sizes of rays affect iNeRF on a synthetic dataset. We then show that for complex real-world scenes from the LLFF dataset [21], iNeRF can improve NeRF by estimating the camera poses of novel images and using these images as additional training data for NeRF. Finally, we show iNeRF can perform categorylevel object pose estimation, including object instances not seen during training, with RGB images by inverting a NeRF model inferred from a single view.

Subfields: Aerospace Engineering, Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/3882aa4b2370cae3f8b7b9c41e9088f8c056cdd9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Vision-Only Robot Navigation in a Neural Radiance World (2021, methodology): “As in [19], a subset of pixels I are selected for evaluation using existing image feature detectors (e.”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, unknown): “iNeRF [125] (December 2020) formulated pose recon-”
NICE-SLAM: Neural Implicit Scalable Encoding for SLAM (2021, background): “Another set of works [21, 56, 65] tackle the problem of camera pose optimization, but they need a rather long optimization process, which is not suitable for real-time applications.”
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation (2021, background): “Lastly, neural fields have been leveraged to find unknown camera poses in 3D reconstruction tasks [19, 45].”
NeRF-: Neural Radiance Fields Without Known Camera Parameters (2021, background): “Inparticular, a concurrent work, iNeRF [Yen-Chen et al. 2020], is closely related to ours.”
KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs (2021, background): “A series of works improve generalization from limited number of training views [13,42,44,59,63,66,75] while others [57,68,73] remove the requirement for pose estimation.”
Depth-supervised NeRF: Fewer Views and Faster Training for Free (2021, methodology): “Neural rendering with implicit representations has become a widely-used technique for solving many vision and graphics tasks ranging from view synthesis [5, 15, 25], to re-lighting [12,13], to pose and shape estimation [17,21,31], to 3D-aware image synthesis and editing [3, 11, 23], to modeling dyna”
Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction (2021, background): “, relighting [2, 3, 50, 70], deformation [13, 38–40, 55], self-calibration [19, 27, 28, 35, 61], meta-learning [52], dynamic scene modeling [14, 25, 33, 41, 64], and generative modeling [5, 22, 44].”
Block-NeRF: Scalable Large Scene Neural View Synthesis (2022, methodology): “Our model also incorporates a learned camera pose refinement which has been explored in previous works [32, 58, 66, 70, 71].”
Neural RGB-D Surface Reconstruction (2021, background): “Others have focused on resectioning a camera given a learned NeRF [85], and optimizing for the camera poses while learning a NeRF [41, 78].”
A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose (2021, background): “Some concurrent works also use neural scene representations for reﬁning camera motion [75], video reenactment [50], and facial models [15, 16].”
NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields (2022, unknown): “…enticing for building NeRFs without having to process the data to get the camera poses of the images, a task that is usually long and is typically done using COLMAP[24]. iNeRF [25] was the first to show that it is possible to regress the camera pose given an already built NeRF of the scene.”
BARF: Bundle-Adjusting Neural Radiance Fields (2021, methodology): “For evaluating view synthesis, we run an additional step of test-time photometric optimization on the trained models [26, 65] to factor out the pose error that may contaminate the view synthesis quality.”
iMAP: Implicit Mapping and Positioning in Real-Time (2021, background): “Two recent papers [35, 39] have also explored camera pose optimisation.”