Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

By Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, Ren Ng. 2020

3,689 citations recorded by the source corpus.

We show that passing input points through a simple Fourier feature mapping enables a multilayer perceptron (MLP) to learn high-frequency functions in low-dimensional problem domains. These results shed light on recent advances in computer vision and graphics that achieve state-of-the-art results by using MLPs to represent complex 3D objects and scenes. Using tools from the neural tangent kernel (NTK) literature, we show that a standard MLP fails to learn high frequencies both in theory and in practice. To overcome this spectral bias, we use a Fourier feature mapping to transform the effective NTK into a stationary kernel with a tunable bandwidth. We suggest an approach for selecting problem-specific Fourier features that greatly improves the performance of MLPs for low-dimensional regression tasks relevant to the computer vision and graphics communities.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Model Architectures

Citations in the corpus

Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, methodology): “1), we use a 2D Fourier feature positional encoding (Vaswani et al., 2017; Stanley, 2007; Mildenhall et al., 2020; Tancik et al., 2020) using a sine and cosine bands with frequencies spaced linearly from a minimum frequency to a maximum frequency.”
Variational Diffusion Models (2021, neutral): “Previous work [Tancik et al., 2020] has used these features for input coordinates to model high frequency details across the spatial dimension, and for time embeddings to condition denoising networks over the temporal dimension[Song et al.”
Score-Based Generative Modeling through Stochastic Differential Equations (2020, positive): “The classiﬁer is conditioned on log σ i using random Fourier features (Tancik et al., 2020), and the training objective is a simple sum of cross-entropy losses sampled at different scales.”
Segment Anything (2023, methodology): “We represent points and boxes by positional encodings [95] summed with learned embeddings for each prompt type and free-form text with an off-the-shelf text encoder from CLIP [82].”
Elucidating the Design Space of Diffusion-Based Generative Models (2022, methodology): “Second, DDPM++ inherits its positional encoding scheme for the noise level directly from DDPM [15] whereas NCSN++ replaces it with random Fourier features [45].”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, methodology): “The synthesis network G of StyleGAN3 starts from a spatial map defined by Fourier features [Tancik et al. 2020; Xu et al. 2021].”
Neural Additive Models: Interpretable Machine Learning with Neural Nets (2020, background): “For example, fourier features [43] have been shown to be highly effective for learning high frequency functions via neural networks and might be useful for training expressive NAMs.”
Alias-Free Generative Adversarial Networks (2021, methodology): “Recent implicit network [51, 54, 13] based GANs [4, 52] generate each pixel independently via similar 1×1 convolutions.”
FastNeRF: High-Fidelity Neural Rendering at 200FPS (2021, methodology): “After use in [41], this was popularized in [25, 49] and explored in greater detail by [40].”
LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (2024, methodology): “This dimensional monotonicity is based on the NTK theory (Jacot et al., 2018; Tancik et al., 2020; LocalLLaMA, 2023b), suggesting that lower dimensions with higher frequency requires less interpolation (i.e., a smaller λ i ), and higher dimensions with lower frequency can do more interpolation…”
Text2Mesh: Text-Driven Neural Stylization for Meshes (2021, methodology): “contain high-frequency content with high fidelity, we use a frequency-based positional encoding [56].”
Learned Initializations for Optimizing Coordinate-Based Neural Representations (2020, methodology): “For the MLP fθ, we use 5 layers with 256 channels each and ReLU nonlinearities, and we apply random Fourier features to the input coordinates [36].”
NeuMan: Neural Human Radiance Field from a Single Video (2022, positive): “The radiance field function F Θ is often implemented with multi-layer perceptrons (MLPs) with positional encodings [43,27] and periodic activations [2].”
MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction (2022, background): “After their introduction to novel view synthesis [44], positional encoding functions are now widely used for neural implicit surface reconstruction [49,76,81,82] as they increase the expressiveness of coordinate-based networks [70].”
Instant neural graphics primitives with a multiresolution hash encoding (2022, methodology): “The five dimensions of this light field are independently encoded using the above formula; this was later extended to randomly oriented parallel wavefronts [Tancik et al. 2020] and level-of-detail filtering [Barron et al.”
NeRP: Implicit Neural Representation Learning with Prior Embedding for Sparsely Sampled Image Reconstruction (2021, methodology): “1) Fourier Feature Embedding: Since Fourier features are shown to be effective for networks to learn high-frequency functions [20], we use a Fourier feature mapping γ to encode the input coordinates c before applying them to the coordinate-based network.”
Protein structure generation via folding diffusion (2022, methodology): “To incorporate the timestep t into this model, we generate random Fourier feature embeddings (Tancik et al., 2020) as done in Song et al.”