Implicit Neural Representations with Periodic Activation Functions

By Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, Gordon Wetzstein. 2020

3,729 citations recorded by the source corpus.

Implicitly defined, continuous, differentiable signal representations parameterized by neural networks have emerged as a powerful paradigm, offering many possible benefits over conventional representations. However, current network architectures for such implicit neural representations are incapable of modeling signals with fine detail, and fail to represent a signal's spatial and temporal derivatives, despite the fact that these are essential to many physical signals defined implicitly as the solution to partial differential equations. We propose to leverage periodic activation functions for implicit neural representations and demonstrate that these networks, dubbed sinusoidal representation networks or Sirens, are ideally suited for representing complex natural signals and their derivatives. We analyze Siren activation statistics to propose a principled initialization scheme and demonstrate the representation of images, wavefields, video, sound, and their derivatives. Further, we show how Sirens can be leveraged to solve challenging boundary value problems, such as particular Eikonal equations (yielding signed distance functions), the Poisson equation, and the Helmholtz and wave equations. Lastly, we combine Sirens with hypernetworks to learn priors over the space of Siren functions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

How to Represent Part-Whole Hierarchies in a Neural Network (2021, methodology): “Work on the use of neural ﬁelds for generating images has established that there are much better ways to represent the location than using two scalars for its x and y coordinates [Sitzmann et al., 2020, Mildenhall et al., 2020].”
Alias-Free Generative Adversarial Networks (2021, methodology): “Recent implicit network [51, 54, 13] based GANs [4, 52] generate each pixel independently via similar 1×1 convolutions.”
BigVGAN: A Universal Neural Vocoder with Large-Scale Training (2022, methodology): “Inspired by the methods proposed for other domains (Liu et al., 2020; Sitzmann et al., 2020), we demonstrate the noticeable success of periodic activations in audio synthesis.”
On Embeddings for Numerical Features in Tabular Deep Learning (2022, neutral): “Examples include NLP [45], CV [25], implicit neural representations [28, 38, 42].”
NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior (2022, methodology): “Recently, SiNeRF [48] uses SIREN [36] layers and a novel sampling strategy to alleviate the sub-optimality of joint optimisation in NeRFmm.”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “continuous signals, mapping coordinates to data values [196], [210].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, neutral): “The generator was a SIREN-based [115] NeRF volumetric renderer, with”
WIRE: Wavelet Implicit Neural Representations (2023, methodology): “This has been remedied by several modifications to the MLP including the so-called positional encoding [28, 43], as well as various choices of nonlinearity such as the sinusoidal function [41] and the Gaussian function [33].”
CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Method (2021, neutral): “The subsequent hidden layers also use “sine” activation, and their weights are initialized by He uniform distribution [52,53].”
HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation (2020, methodology): “text analysis [13, 50] to 3D modeling [26, 42], but rarely for generating image-like maps.”
StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation (2021, background): “The volume rendering network contains eight shared modulated FC layers with SIREN [62] activation:”
AutoInt: Automatic Integration for Fast Neural Volume Rendering (2020, background): “Right: a 1D scanline of the sinogram shows that Swish interpolates missing data best while sine activations [51] tend to overfit the measurements.”
Multi-View Mesh Reconstruction with Neural Deferred Shading (2022, positive): “In Figure 9, we show the different results using positional encoding (PE) [69], Gaussian Fourier features (GFF) [69], sinusoidal activations (SIREN) [66] and a standard MLP with ReLU activations.”
You Only Learn One Representation: Unified Network for Multiple Tasks (2021, methodology): “The methods that belong to the category of implicit deep learning are mainly implicit neural representations [11] and deep equilibrium models [2, 3, 19].”