NeX: Real-time View Synthesis with Neural Basis Expansion

By Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, Supasorn Suwajanakorn. Computer Vision and Pattern Recognition, 2021

355 citations recorded by the source corpus.

We present NeX, a new approach to novel view synthesis based on enhancements of multiplane image (MPI) that can reproduce next-level view-dependent effects—in real time. Unlike traditional MPI that uses a set of simple RGBα planes, our technique models view-dependent effects by instead parameterizing each pixel as a linear combination of basis functions learned from a neural network. Moreover, we propose a hybrid implicit-explicit modeling strategy that improves upon fine detail and produces state-of-the-art results. Our method is evaluated on benchmark forwardfacing datasets as well as our newly-introduced dataset designed to test the limit of view-dependent modeling with significantly more challenging effects such as the rainbow reflections on a CD. Our method achieves the best overall scores across all major metrics on these datasets with more than 1000× faster rendering time than the state of the art. For real-time demos, visit https://nex-mpi.github.io/

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0283d3ace44c0deeb9c56039989eeb072c54f825

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures (2022, background): “Storage can be reduced by interpolating intermediate images with optical flow [5], representing the light field as a neural network [1], or by reconstructing a Multi-Plane Image (MPI) representation of the scene [15,32,37,47,54].”
Plenoxels: Radiance Fields without Neural Networks (2021, methodology): “Spherical harmonics are a standard basis for functions over the sphere, and have been used previously to represent view dependence [4, 36, 47, 58, 59].”
Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields (2021, positive): “Recent works have used volumetric representations such as voxel grids [18] or multiplane images [9,21,35,39,46], which are better suited for gradient-based optimization than meshes.”
FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting (2023, neutral): “The pseudo views sampling is enabled after 2,000 iterations to ensure the Gaussians can roughly represent the scene.”
NeRF-Editing: Geometry Editing of Neural Radiance Fields (2022, background): “However, NeRF still has shortcomings and plenty of work has extended the original NeRF, including better synthesis effects [36, 71, 83], applicable to dynamic scenes [14, 33, 44–48, 67, 72, 82], faster rendering speed [18, 22, 49, 77], generalization to different scenes [8, 69], relighting [4, 60, 8”
MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes (2023, background): “Similarly, multi-plane image [Flynn et al. 2019; Mildenhall et al. 2019; Wizadwongsa et al. 2021; Zhou et al. 2018] or multi-sphere image [Attal et al.”
HeadNeRF: A Realtime NeRF-based Parametric Head Model (2021, background): “Besides, many works are proposed to further improve NeRF, including speeding up training [1,2] and inference [13,18,44,56,59], improving rendering quality [55], and reducing the number of required inputs [8, 60].”
SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis (2023, positive): “proaches [47, 38, 22, 43, 15, 32, 16, 28] attempted to speed up the standard NeRF.”
Advances in Neural Rendering (2021, methodology): “…to mitigating the memory challenges of dense voxel grids include using object-speciﬁc shape templates [KTEM18], multi-plane [ZTF ∗ 18, MSOC ∗ 19, FBD ∗ 19, TS20, WPYS21] or multi-sphere [BFO ∗ 20, ALG ∗ 20] images, which all aim at representing the voxel grid using a sparse approximation.”
Instant neural graphics primitives with a multiresolution hash encoding (2022, background): “At HD resolutions, synthetic and even real-world scenes can be trained in seconds and rendered at 60 FPS, without the need of caching of the MLP outputs [Garbin et al. 2021; Wizadwongsa et al. 2021; Yu et al. 2021b].”
BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis (2023, methodology): “…mesh reconstruction by using volumetric representations of geometry and appearance, such as voxel grids [Lombardi et al. 2019; Penner and Zhang 2017; Szeliski and Golland 1999; Vo-giatzis et al. 2007] or multiplane images [Srinivasan et al. 2019; Wizadwongsa et al. 2021; Zhou et al. 2018].”
K-Planes: Explicit Radiance Fields in Space, Time, and Appearance (2023, methodology): “Instead, we replace the SH functions with a learned basis, retaining the interpretability of treating features as coefficients for a linear decoder yet increasing the expressivity of the basis and allowing it to adapt to each scene, as was proposed in NeX [36].”
PlenOctrees for Real-time Rendering of Neural Radiance Fields (2021, background): “Concurrent with this work, NeX [59] extends MPIs to encode spherical basis functions that enable view-dependent rendering effects in real-time.”