Stereo magnification

By Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, Noah Snavely. ACM Transactions on Graphics, 2018

430 citations recorded by the source corpus.

The view synthesis problem---generating novel views of a scene from known imagery---has garnered recent attention due in part to compelling applications in virtual and augmented reality. In this paper, we explore an intriguing scenario for view synthesis: extrapolating views from imagery captured by narrow-baseline stereo cameras, including VR cameras and now-widespread dual-lens camera phones. We call this problem stereo magnification, and propose a learning framework that leverages a new layered representation that we call multiplane images (MPIs). Our method also uses a massive new data source for learning view extrapolation: online videos on YouTube. Using data mined from such videos, we train a deep network that predicts an MPI from an input stereo image pair. This inferred MPI can then be used to synthesize a range of novel views of the scene, including views that extrapolate significantly beyond the input baseline. We show that our method compares favorably with several recent view synthesis methods, and demonstrate applications in magnifying narrow-baseline stereo images.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/907e0f3300f2da386150fcdae3e0a7623db887de

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis (2023, methodology): “…mesh reconstruction by using volumetric representations of geometry and appearance, such as voxel grids [Lombardi et al. 2019; Penner and Zhang 2017; Szeliski and Golland 1999; Vo-giatzis et al. 2007] or multiplane images [Srinivasan et al. 2019; Wizadwongsa et al. 2021; Zhou et al. 2018].”
Immersive light field video with a layered mesh representation (2020, methodology): “Our analysis is similar to conditions for the spacing of MPI planes proposed in [Flynn et al. 2019; Zhou et al. 2018] and further refined using Fourier analysis in [Mildenhall et al. 2019].”
TensoRF: Tensorial Radiance Fields (2022, background): “Many neural representations [50,35,20,3] are proposed for high-quality rendering.”
MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes (2023, background): “Similarly, multi-plane image [Flynn et al. 2019; Mildenhall et al. 2019; Wizad-wongsa et al. 2021; Zhou et al. 2018] or multi-sphere image [Attal et al. 2020; Broxton et al. 2020] representations map well to graphics hardware and can be rendered in real-time, but also support only restricted camera…”
Dense Depth Priors for Neural Radiance Fields from Sparse Input Views (2021, neutral): “More recently, these mesh-based geometry models have been replaced by volumetric scene representations such as voxel grids [18] or multiplane images [11, 20, 24, 29].”
Light Field Super-Resolution with Zero-Shot Learning (2021, methodology): “Instead of feeding the light field into the network directly, we generate a plane-sweep volume (PSV), which is proved to be efficient for scene geometry inference [25, 26, 27, 51], as the input of AlignNet.”
Deferred neural rendering (2019, background): “In addition we provide a background image to the neural renderer, to output the final reenactment result. warping [Park et al. 2017; Zhou et al. 2016], or a layered scene representation [Tulsiani et al. 2018; Zhou et al. 2018].”
NeX: Real-time View Synthesis with Neural Basis Expansion (2021, methodology): “[53] is a scene representation that consists of parallel semitransparent planes placed along a reference viewing frustum.”
Learning the Depths of Moving People by Watching Frozen People (2019, methodology): “[53], we use ORB-SLAM2 [24] to identify trackable sequences in each video and to estimate an initial camera pose for each frame.”
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields (2021, background): “Early approaches directly color voxel grids using observed images [37], and more recent volumetric approaches use gradient-based learning to train deep networks to predict voxel grid representations of scenes [12, 25, 29, 38, 41, 53].”
Local light field fusion (2019, positive): “This ablation can be considered an upper bound on the performance of Zhou et al. [2018], since we use one MPI generated by a higher capacity 3D CNN.”
Point-NeRF: Point-based Neural Radiance Fields (2022, background): “Recently, various neural scene representations have been presented [4, 29, 45, 61], advancing the state of the art in novel view synthesis and realistic rendering, with volumetric neural radiance fields (NeRFs) [34] producing high fidelity results.”
IBRNet: Learning Multi-View Image-Based Rendering (2021, methodology): “RealEstate10K is a large dataset of indoor scenes with camera poses derived from about 80K video clips, where the cameras lie on 1D camera trajectories.”
NeRF-Editing: Geometry Editing of Neural Radiance Fields (2022, background): “It adopts deep neural networks to synthesize images, which can be employed on multiple representations, such as voxels [39, 54], point clouds [1, 10], meshes [7,50,51,66], multi-plane images (MPIs) [34,42,87] and implicit fields [31, 55].”