IBRNet: Learning Multi-View Image-Based Rendering

By Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P. Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, Thomas A. Funkhouser. Computer Vision and Pattern Recognition, 2021

985 citations recorded by the source corpus.

We present a method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views. The core of our method is a network architecture that includes a multilayer perceptron and a ray transformer that estimates radiance and volume density at continuous 5D locations (3D spatial locations and 2D viewing directions), drawing appearance information on the fly from multiple source views. By drawing on source views at render time, our method hearkens back to classic work on image-based rendering (IBR), and allows us to render high-resolution imagery. Unlike neural scene representation work that optimizes per-scene functions for rendering, we learn a generic view interpolation function that generalizes to novel scenes. We render images using classic volume rendering, which is fully differentiable and allows us to train using only multi-view posed images as supervision. Experiments show that our method outperforms recent novel view synthesis methods that also seek to generalize to novel scenes. Further, if fine-tuned on each scene, our method is competitive with state-of-the-art single-scene neural rendering methods.1

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/7cbc3dd0280b8c4551ac934af42dc227d43754f7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, methodology): “Previous methods (Wang et al., 2021; Chen et al., 2021; Long et al., 2022; Reizenstein et al., 2021; Trevithick & Yang, 2021) have tried to learn data priors so as to infer NeRF from a sparse set of images.”
Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields (2023, background): “IBRNet [43], MVSNeRF [6], and PixelNeRF [48] construct a generalizable 3D representation by leveraging features gathered from various observed viewpoints.”
HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video (2022, background): “Previous neural rendering methods [3, 33, 36, 37, 50, 66, 75] typically assume multi-view input, careful lab capture, or do not perform well on humans due to non-rigid body motion.”
Generative Novel View Synthesis with 3D-Aware Diffusion Models (2023, methodology): “[81] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T.”
iNeRF: Inverting Neural Radiance Fields for Pose Estimation (2020, methodology): “Although NeRF originally needs to optimize the representation for every scene independently, several extensions [28], [39], [45], [47] have been proposed to directly predict a”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, methodology): “PointNeRF outperformed methods such as PixelNeRF [58], MVSNeRF [57], IBRNet [91] after per-scene optimization in the form of point growing and point pruning on the DTU dataset [26].”
HexPlane: A Fast Representation for Dynamic Scenes (2023, background): “NeRF [46] and its variants [2, 3, 44, 48, 74, 76, 85, 92] show impressive results on novel view synthesis [9,80,87,99] and many other applications including 3D reconstruction [42, 72, 90, 94, 100], semantic segmentation [27,59,98], generative model [5,6,10, 49,62,82], and 3D content creation…”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, neutral): “The ﬁnal aspect of driving simulation pertains to ensuring that the simulated vehicle adheres to physically plausible motion.”
ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes (2023, methodology): “In addition to benchmarking NVS methods that are optimized on a per-scene level, we believe that the scale and diversity of ScanNet++ will encourage research on NVS methods that generalize over multiple scenes [51, 47, 6, 19, 53, 41, 26].”
Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans (2020, methodology): “[28, 23, 7, 62, 31, 30, 63] replace hand-crafted parts of the image-based rendering pipeline with learnable counterparts to improve the robustness.”
RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs (2021, background): “Sparse Input Novel-View Synthesis: One approach for circumventing the requirement of dense inputs is to aggregate prior knowledge by pre-training a conditional model of radiance ﬁelds [6,8,20,26,29,47,56,58,62].”
KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs (2021, background): “A series of works improve generalization from limited number of training views [13,42,44,59,63,66,75] while others [57,68,73] remove the requirement for pose estimation.”
NeRF-Editing: Geometry Editing of Neural Radiance Fields (2022, background): “However, NeRF still has shortcomings and plenty of work has extended the original NeRF, including better synthesis effects [36, 71, 83], applicable to dynamic scenes [14, 33, 44–48, 67, 72, 82], faster rendering speed [18, 22, 49, 77], generalization to different scenes [8, 69], relighting [4, 60, 8”
ReconFusion: 3D Reconstruction with Diffusion Priors (2023, positive): “These networks leverage large collections of posed multiview data across many scenes [8, 11, 19, 21, 44, 55, 57, 60, 67, 71].”
Nerfstudio: A Modular Framework for Neural Radiance Field Development (2023, methodology): “including few-image training [Wang et al. 2021b; Yu et al. 2021], explicit features for editing [Liu et al.”
Splatter Image: Ultra-Fast Single-View 3D Reconstruction (2023, background): “When more than one view is available at the input, one can learn to interpolate between available views in the 3D space to estimate the scene geometry [5, 21], learn a view interpolation function [41] or optimize a 3D representation of a scene using semantic priors [11].”
Baking Neural Radiance Fields for Real-Time View Synthesis (2021, methodology): “…[11] 33.00 0.962 0.038 300 0.01 0.00002 NeRF [29] 31.00 0.947 0.081 300 0.03 0.00011 JAXNeRF [11] 31.65 0.952 0.051 300 0.05 0.00016 IBRNet [43] 28.14 0.942 0.072 300 0.18 0.00061 AutoInt [23] 25.55 0.911 0.170 300 0.38 0.00128 NSVF [24] 31.74 0.953 0.047 300 0.65 0.00217 NV [25] 26 is an…”
TensoRF: Tensorial Radiance Fields (2022, methodology): “Other methods design generalizable network modules trained across scenes to achieve image-dependent radiance field rendering [48,41,8] and fast reconstruction [7,43].”