BARF: Bundle-Adjusting Neural Radiance Fields

By Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, Simon Lucey. IEEE International Conference on Computer Vision, 2021

894 citations recorded by the source corpus.

Neural Radiance Fields (NeRF) [31] have recently gained a surge of interest within the computer vision community for its power to synthesize photorealistic novel views of real-world scenes. One limitation of NeRF, however, is its requirement of accurate camera poses to learn the scene representations. In this paper, we propose Bundle-Adjusting Neural Radiance Fields (BARF) for training NeRF from imperfect (or even unknown) camera poses — the joint problem of learning neural 3D representations and registering camera frames. We establish a theoretical connection to classical image alignment and show that coarse-to-fine registration is also applicable to NeRF. Furthermore, we show that naïvely applying positional encoding in NeRF has a negative impact on registration with a synthesis-based objective. Experiments on synthetic and real-world data show that BARF can effectively optimize the neural scene representations and resolve large camera pose misalignment at the same time. This enables view synthesis and localization of video sequences from unknown camera poses, opening up new avenues for visual localization systems (e.g. SLAM) and potential applications for dense 3D mapping and reconstruction.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/33cc02f23c97a3daa835953b9d2784d0e1abf16e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

DUSt3R: Geometric 3D Vision Made Easy (2023, neutral): “These improvements encompass advanced feature description [26, 28, 97, 135, 167], more accurate image matching [3, 17, 60, 82, 100, 120, 126, 145], featuremetric refinement [59], and neural bundle adjustment [58, 153].”
FreeNeRF: Improving Few-Shot Neural Rendering with Free Frequency Regularization (2023, background): “Recent works show the beneﬁts of gradually increasing the input frequency in different applications, such as non-rigid scene deformation [23], bundle adjustment [16], surface reconstruction [33], and ﬁtting functions with a wider frequency band [9].”
NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images (2021, methodology): “This could potentially be addressed by jointly optimizing RawNeRF and the input camera poses [33, 46].”
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation (2021, background): “Lastly, neural fields have been leveraged to find unknown camera poses in 3D reconstruction tasks [19, 45].”
Fast Dynamic Radiance Fields with Time-Aware Neural Voxels (2022, background): “2021], camera parameter optimization [Lin et al. 2021; Wang et al. 2021b], rendering large-scale unbounded scenes [Zhang et al.”
Nerfstudio: A Modular Framework for Neural Radiance Field Development (2023, methodology): “The Nerfacto method first optimizes camera views using an optimized SE(3) transformation [Lin et al. 2021; Tancik et al. 2022; Wang et al. 2021c].”
NeuMan: Neural Human Radiance Field from a Single Video (2022, background): “While originally proposed to reconstruct a static scene with a set of posed images, it has since been quickly extended to dynamic scenes [29,15,30] and uncalibrated scenes [45,17].”
BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering (2021, background): “[26,13,25] alternatively adopt windowed positional encoding to aid learning dynamic shapes via a coarse-to-fine training.”
NICE-SLAM: Neural Implicit Scalable Encoding for SLAM (2021, background): “Another set of works [21, 56, 65] tackle the problem of camera pose optimization, but they need a rather long optimization process, which is not suitable for real-time applications.”
DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes (2023, neutral): “We also find that our baseline method, 3DGS [4], fails in modeling the detail areas, producing unpleasant artifacts, blurring, and unnatural colors.”
Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (2023, background): “…through the use of MLPs. Subsequent research has expanded the utility of NeRF to various applications, encompassing tasks such as mesh reconstruction from a collection of images [20, 45], inverse rendering [5, 25, 54], optimization of camera parameters [21, 47, 48], and few-shot learning [10, 51].”
Urban Radiance Fields (2021, background): “First, it assumes that good camera parameters are given through an SfM pipeline, but they are sometimes noisy in practice — a joint optimization of the camera parameters along with the network parameters may improve the reconstructions for those cases [37, 67].”
NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior (2022, methodology): “BARF [4] uses test-time optimisation to identify poses for the test frames, where all poses are initialised with identity matrices.”
Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction (2021, background): “, relighting [2, 3, 50, 70], deformation [13, 38–40, 55], self-calibration [19, 27, 28, 35, 61], meta-learning [52], dynamic scene modeling [14, 25, 33, 41, 64], and generative modeling [5, 22, 44].”
SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis (2023, background): “NeRF −− [56], GNeRF [33], and BARF [31] relaxed the requirements of NeRFs and synthesized novel views without perfect camera poses.”
Neural RGB-D Surface Reconstruction (2021, background): “Others have focused on resectioning a camera given a learned NeRF [85], and optimizing for the camera poses while learning a NeRF [41, 78].”