Immersive light field video with a layered mesh representation

By Michael Broxton, John Flynn, Ryan S. Overbeck, Daniel Erickson, Peter Hedman, Matthew DuVall, Jason Dourgarian, Jay Busch, Matt Whalen, Paul E. Debevec. ACM Transactions on Graphics, 2020

330 citations recorded by the source corpus.

We present a system for capturing, reconstructing, compressing, and rendering high quality immersive light field video. We accomplish this by leveraging the recently introduced DeepView view interpolation algorithm, replacing its underlying multi-plane image (MPI) scene representation with a collection of spherical shells that are better suited for representing panoramic light field content. We further process this data to reduce the large number of shell layers to a small, fixed number of RGBA+depth layers without significant loss in visual quality. The resulting RGB, alpha, and depth channels in these layers are then compressed using conventional texture atlasing and video compression techniques. The final compressed representation is lightweight and can be rendered on mobile VR/AR platforms or in a web browser. We demonstrate light field video results using data from the 16-camera rig of [Pozo et al. 2019] as well as a new low-cost hemispherical array made from 46 synchronized action sports cameras. From this data we produce 6 degree of freedom volumetric videos with a wide 70 cm viewing baseline, 10 pixels per degree angular resolution, and a wide field of view, at 30 frames per second video frame rates. Advancing over previous work, we show that our system is able to reproduce challenging content such as view-dependent reflections, semi-transparent surfaces, and near-field objects as close as 34 cm to the surface of the camera rig.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8d7ec17c81301d8c8c2e2013c5d4bfc14f4c5e94

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Enhancing Photorealism Enhancement (2021, background): “real imagery of a scene for rendering novel views [36], [37], [38], [39], [40], [41], [42], [43], [44].”
Advances in Neural Rendering (2021, methodology): “…to mitigating the memory challenges of dense voxel grids include using object-speciﬁc shape templates [KTEM18], multi-plane [ZTF ∗ 18, MSOC ∗ 19, FBD ∗ 19, TS20, WPYS21] or multi-sphere [BFO ∗ 20, ALG ∗ 20] images, which all aim at representing the voxel grid using a sparse approximation.”
AutoInt: Automatic Integration for Fast Neural Volume Rendering (2020, background): “Neural scene representations often use an explicit 3D proxy geometry, such as multi-plane [15, 37, 64] or multi-sphere [2, 4] images or a voxel grid of features [32, 54].”
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes (2020, background): “Yet another promising 3D representation is multiplane images (MPIs), that have been shown to model complex scene appearance [9, 10, 15, 20, 45, 69].”
Dynamic View Synthesis from Dynamic Monocular Video (2021, methodology): “Several systems have been proposed to support interactive viewpoint control watching videos of generic scenes [65, 11, 41, 4, 5, 1].”
Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (2023, methodology): “[10] use multi-sphere image as bootstrap and then convert it to layered meshes.”
MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes (2023, background): “…multi-plane image [Flynn et al. 2019; Mildenhall et al. 2019; Wizad-wongsa et al. 2021; Zhou et al. 2018] or multi-sphere image [Attal et al. 2020; Broxton et al. 2020] representations map well to graphics hardware and can be rendered in real-time, but also support only restricted camera motion.”
Neural 3D Video Synthesis from Multi-view Video (2021, neutral): “We also demonstrate the generality of our method using the multi-view videos from [6] directly trained on their fisheye video input.”
Mixture of volumetric primitives for efficient neural rendering (2021, background): “Multi-sphere images (MSIs) [Attal et al. 2020; Broxton et al. 2020] replace the planar and textured geometry proxies with a set of textured concentric sphere proxies.”
Baking Neural Radiance Fields for Real-Time View Synthesis (2021, neutral): “[4]), 360 ◦ scenes represent a challenging and general use-case that has not yet been addressed.”
NeRF++: Analyzing and Improving Neural Radiance Fields (2020, background): “…our inverted sphere parameterization is related to the concept of multi-sphere images (a scene representation consisting of nested concentric spheres sampled according to inverse depth from the sphere centers) proposed in recent work on view synthesis (Attal et al., 2020; Broxton et al., 2020).”
Space-time Neural Irradiance Fields for Free-Viewpoint Video (2020, methodology): “Consequently, most of the existing methods typical require laborious multi-camera setup [78, 11, 52, 13, 6], special hardware [1], or synchronous video captures from multiple viewpoints [3, 4].”
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (2023, neutral): “3.2.”
MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures (2022, background): “Multi-sphere images enable larger fields of view [2,6], but these representations still only support limited output camera motion”