Neural Networks Fail to Learn Periodic Functions and How to Fix It

By Liu Ziyin, Tilman Hartwig, Masahito Ueda. 2020

233 citations recorded by the source corpus.

Previous literature offers limited clues on how to learn a periodic function using modern neural networks. We start with a study of the extrapolation properties of neural networks; we prove and demonstrate experimentally that the standard activations functions, such as ReLU, tanh, sigmoid, along with their variants, all fail to learn to extrapolate simple periodic functions. We hypothesize that this is due to their lack of a "periodic" inductive bias. As a fix of this problem, we propose a new activation, namely, $x + \sin^2(x)$, which achieves the desired periodic inductive bias to learn a periodic function while maintaining a favorable optimization property of the ReLU-based activations. Experimentally, we apply the proposed method to temperature and financial data prediction.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Statistical and Nonlinear Physics

Citations in the corpus

StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “We employ the snake activation function [46], proven effective for waveform generation in [31].”
High-Fidelity Audio Compression with Improved RVQGAN (2023, methodology): “However, our recipe has the following key differences: 1) We introduce a periodic inductive bias using Snake activations [46, 21] 2) We improve codebook learning by projecting the encodings into a low-dimensional space [43] 3) We obtain a stable training recipe using best practices for adversarial a”
Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (2023, background): “• Vocos with Snake activation : Although Snake (Ziyin et al., 2020) has been shown to enhance time-domain vocoders such as BigVGAN, in our case, it did not result in performance gains; in fact, it showed a slight decline.”
BigVGAN: A Universal Neural Vocoder with Large-Scale Training (2022, methodology): “Our generator has a connection with the results in time-series prediction [27] and image synthesis [17].”
Can deep learning beat numerical weather prediction? (2021, unknown): “As shown in Ziyin et al. [99], NNs generally have difﬁculties extrapolating periodic features correctly.”