Free View Synthesis

By Gernot Riegler, Vladlen Koltun. European Conference on Computer Vision, 2020

388 citations recorded by the source corpus.

We present a method for novel view synthesis from input images that are freely distributed around a scene. Our method does not rely on a regular arrangement of input views, can synthesize images for free camera movement through the scene, and works for general scenes with unconstrained geometric layouts. We calibrate the input images via SfM and erect a coarse geometric scaffold via MVS. This scaffold is used to create a proxy depth map for a novel view of the scene. Based on this depth map, a recurrent encoder-decoder network processes reprojected features from nearby views and synthesizes the new view. Our network does not need to be optimized for a given scene. After training on a dataset, it works in previously unseen environments with no fine-tuning or per-scene optimization. We evaluate the presented approach on challenging real-world datasets, including Tanks and Temples, where we demonstrate successful view synthesis for the first time and substantially outperform prior and concurrent work.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/49fae04a4e9383080788759f63dba75c86bd21b0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis (2023, methodology): “While current NeRF-like approaches are able to recover high-quality real-time-renderable meshes of individual objects with simple geometry [Boss et al. 2022], reconstructing detailed and well-behaved meshes from captures of real-world unbounded scenes (such as the “360 degree captures” of Barron et…”
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes (2020, background): “Such methods have learned additional deep features that exist on top of reconstructed meshes [25, 60, 73] or dense depth maps [21, 79].”
NeRF-Editing: Geometry Editing of Neural Radiance Fields (2022, background): “It adopts deep neural networks to synthesize images, which can be employed on multiple representations, such as voxels [39, 54], point clouds [1, 10], meshes [7,50,51,66], multi-plane images (MPIs) [34,42,87] and implicit fields [31, 55].”
Space-time Neural Irradiance Fields for Free-Viewpoint Video (2020, background): “Existing image-based rendering techniques first extract approximated geometric proxy and create novel target views by warping and blending the corresponding contents from multiple source frames [28, 55, 22, 57].”
Editing Conditional Radiance Fields (2021, background): “Instead, several recent approaches implicitly encode a continuous volumetric representation of shape [14, 21, 38, 39, 42, 43, 48, 50, 54, 61] or both shape and view-dependent appearance [41, 45, 47, 59, 63, 66, 69, 76, 12, 73] in the weights of a neural network.”
Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video (2020, background): “Most recent work is designed for static scenes [24, 13, 53, 44, 14, 69, 70, 46, 63]; methods for dynamic scenes are in their infancy.”
NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images (2021, background): “A more feasible capture scenario is to capture more widely spaced inputs and use a “proxy” geometry ( e.g ., a reconstructed tri-angle mesh) to reproject and combine colors from the input images, using either a heuristic [6] or learned [21, 39, 40] blending function.”
CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields (2021, background): “The past few years have witnessed tremendous progress in the implicit representation of 3D models with neural networks [9, 14, 16, 23, 27, 34].”
Dynamic View Synthesis from Dynamic Monocular Video (2021, background): “Prior work addresses this via local warps [9], operating in the gradient domain [23], soft 3D reconstruction [41], and learning-based approaches [21, 15, 14, 20, 44].”
KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D (2021, methodology): “based novel view synthesis, including methods based on Neural Radiance Fields [7], [27], [64] or per-view depth maps [54], [83].”
Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation (2022, methodology): “of generating segmentation images for any arbitrary view is to first synthesize color image from a desired view using view synthesis methods like [1, 25, 33, 46]; followed by 2D image segmentation [6, 64].”
GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds (2021, methodology): “[52] also used a mesh obtained by multi-view stereo as a coarse input geometry.”
BARF: Bundle-Adjusting Neural Radiance Fields (2021, background): “Various representations have been explored in this direction, e.g . depth [15, 61, 42, 43], layered depth [56, 46], multi-plane images [71, 51, 55], volume density [27, 31], and mesh sheets [20].”
A Survey on 3D Gaussian Splatting (2024, neutral): “Afterwards, point densification and pruning are adopted to control the density of 3D Gaussians.”
Generative Novel View Synthesis with 3D-Aware Diffusion Models (2023, methodology): “[52] Gernot Riegler and Vladlen Koltun.”
Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis (2021, neutral): “Free View Synthesis [35] uses geometric approaches to improve view synthesis in un-bounded in-the-wild scenes.”
UniSim: A Neural Closed-Loop Sensor Simulator (2023, methodology): “FVS [64] is an NVS method that uses reconstructed geometry (aggregated LiDAR in our implementation) as a “proxy” to re-project pixels from the input images into new camera views, where they are blended by a neural network.”
NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior (2022, background): “meshes [30, 31], Multi-Plane Images [41, 59], layered depth [42] etc.”