Learned Initializations for Optimizing Coordinate-Based Neural Representations

By Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, Ren Ng. Computer Vision and Pattern Recognition, 2020

336 citations recorded by the source corpus.

Coordinate-based neural representations have shown significant promise as an alternative to discrete, array-based representations for complex low dimensional signals. However, optimizing a coordinate-based network from randomly initialized weights for each new signal is inefficient. We propose applying standard meta-learning algorithms to learn the initial weight parameters for these fully-connected networks based on the underlying class of signals being represented (e.g., images of faces or 3D models of chairs). Despite requiring only a minor change in implementation, using these learned initial weights enables faster convergence during optimization and can serve as a strong prior over the signal class being modeled, resulting in better generalization when only partial observations of a given signal are available. We explore these benefits across a variety of tasks, including representing 2D images, reconstructing CT scans, and recovering 3D shapes and scenes from 2D image observations.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/eccd3575c9be63b29257a8e63a09b7bbe62fa9fb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

PlenOctrees for Real-time Rendering of Neural Radiance Fields (2021, background): “This can be accomplished by conditioning on predicted images features [51, 61, 56] or meta-learning [49].”
KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs (2021, background): “A series of works improve generalization from limited number of training views [13,42,44,59,63,66,75] while others [57,68,73] remove the requirement for pose estimation.”
FastNeRF: High-Fidelity Neural Rendering at 200FPS (2021, methodology): “It is worth noting that our method does not address training speed, as for example [39].”
COIN: COmpression with Implicit Neural representations (2021, background): “Nevertheless, this limitation could likely be sidestepped with meta-learning [29, 33] or amortized inference [35, 40] approaches.”
Plenoxels: Radiance Fields without Neural Networks (2021, background): “Learned Initializations [51] employs meta-learning on many scenes to start from a better MLP initialization, for both > 10× faster training and better priors when per-scene data is limited.”
Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation (2022, methodology): “Object appearance and shape priors are incorporated via category-specific metalearned initialization [11, 23, 38, 54] of the MLP weights.”
RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs (2021, background): “[54] learn network initializations from which test time optimization on a new scene converges faster.”
Mixture of volumetric primitives for efficient neural rendering (2021, methodology): “…of volumetric models, much work has been dedicated to extending the approach for modeling small motions [Park et al. 2020], illumination variation [Srinivasan et al. 2020], reducing data requirements [Trevithick and Yang 2020; Yu et al. 2020], and learning efficiency [Tancik et al. 2020].”
Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction (2021, background): “, relighting [2, 3, 50, 70], deformation [13, 38–40, 55], self-calibration [19, 27, 28, 35, 61], meta-learning [52], dynamic scene modeling [14, 25, 33, 41, 64], and generative modeling [5, 22, 44].”
Depth-supervised NeRF: Fewer Views and Faster Training for Free (2021, methodology): “PixelNeRF [33] and metaNeRF [26] use data-driven priors recovered from a domain of training scenes to fill in missing information from test scenes.”
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields (2021, background): “NeRF has inspired many subsequent works that extend its continuous neural volumetric representation for generative modeling [8, 36], dynamic scenes [23, 33], nonrigidly deforming objects [13, 34], phototourism settings with changing illumination and occluders [26, 43], and reflectance modeling for r”
Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis (2021, background): “Rather than generalizing between scenes through a shared encoder and decoder, [43, 11] meta-learn radiance field weights that can be adapted to a specific scene in a few gradient steps.”
Editing Conditional Radiance Fields (2021, background): “Instead, several recent approaches implicitly encode a continuous volumetric representation of shape [14, 21, 38, 39, 42, 43, 48, 50, 54, 61] or both shape and view-dependent appearance [41, 45, 47, 59, 63, 66, 69, 76, 12, 73] in the weights of a neural network.”
Generative Novel View Synthesis with 3D-Aware Diffusion Models (2023, methodology): “[77] Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P Srinivasan, Jonathan T Barron, and Ren Ng.”