Neural Network with Unbounded Activation Functions is Universal Approximator

By Sho Sonoda, Noboru Murata. 2015

367 citations recorded by the source corpus.

This paper presents an investigation of the approximation property of neural networks with unbounded activation functions, such as the rectified linear unit (ReLU), which is the new de-facto standard of deep learning. The ReLU network can be analyzed by the ridgelet transform with respect to Lizorkin distributions. By showing three reconstruction formulas by using the Fourier slice theorem, the Radon transform, and Parseval's relation, it is shown that a neural network with unbounded activation functions still satisfies the universal approximation property. As an additional consequence, the ridgelet transform, or the backprojection filter in the Radon domain, is what the network learns after backpropagation. Subject to a constructive admissibility condition, the trained network can be obtained by simply discretizing the ridgelet transform, without backpropagation. Numerical examples not only support the consistency of the admissibility condition but also imply that some non-admissible cases result in low-pass filtering.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

Citations in the corpus

An overview of deep learning in medical imaging focusing on MRI (2018, neutral): “approximate almost any nonlinear function [47,48].”
Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm (2017, background): “The model embedding architecture was the same across all methods, using 4 convolutional layers with 3 × 3 kernels, 64 ﬁlters, stride 2, batch normalization, and ReLU nonlinearities.”
On the Spectral Bias of Neural Networks (2018, background): “This approach has been recently generalized to unbounded activation functions by Sonoda & Murata (2017).”
An Information-Theoretic View for Deep Learning (2018, background): “They have small E[RS(W )] because neural networks with deep architectures can e ciently compactly represent highly-varying functions (Sonoda and Murata 2015).”
On stock return prediction with LSTM networks (2017, background): “10 Sonoda & Murata (2015) investigated the theorem for an unbounded function, since the ReLU function is cutting edge within deep learning, and indeed showed that the theorem still holds for unbounded functions.”
Universal approximations of permutation invariant/equivariant functions by deep neural networks (2019, methodology): “[Sonoda and Murata, 2017], we can approximate φ and ρ by shallow networks.”
A Fine-Grained Spectral Perspective on Neural Networks (2019, background): “Several other works also adopt a spectral perspective on neural networks [13, 55, 19, 7, 66, 74, 67, 68]; here we highlight a few most relevant to us.”
Graph Neural Networks Exponentially Lose Expressive Power for Node Classification (2019, neutral): “…studies have been conducted for Fully Connected Neural Networks (FNNs) (Cybenko, 1989; Hornik, 1991; Hornik et al., 1989; Barron, 1993; Mhaskar, 1993; Sonoda & Murata, 2017; Yarotsky, 2017) and Convolutional Neural Networks (CNNs) (Petersen & Voigtlaender, 2018; Zhou, 2018; Oono & Suzuki, 2019).”
Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality (2018, background): “A remarkable property of neural network is that it has universal approximation capability even if there is only one hidden layer (Cybenko, 1989; Hornik, 1991; Sonoda & Murata, 2015).”
Spiking Neural Networks and Their Applications: A Review (2022, methodology): “…under assumptions on the non-polynomial activation function [14,15]; Sigmoidal activation function and the ReLU are also proved to follow the universal approximation theorem [16]), artiﬁcial neural networks have been widely used as a powerful information-processing tool in machine learning.”
Deep learning classifiers for hyperspectral imaging: A review (2019, methodology): “Despite the wide range of activation functions available in the current DL literature (Agostinelli et al., 2014; Sonoda and Murata, 2017; Ramachandran et al., 2017), the vast majority of DNN models for the analysis of HSI remote sensing data employ ReLU and softmax as the principal activation functi”
Universal Invariant and Equivariant Graph Neural Networks (2019, background): “Note however that universality can be proved for many more non-linearities (Pinkus, 1999) (including the celebrated ReLU function (Sonoda and Murata, 2017)) using different proof technics, and we leave the extension to these cases for future works.”
Geometric deep learning for computational mechanics Part I: Anisotropic Hyperelasticity (2020, methodology): “…with the ReLU activation function (as well as functions similar to that) can be universal approximators for C 1 functions in a Sobolev space (Sonoda and Murata, 2017), our goal here is to directly predict the elastic energy functional by using the Sobolev norm as loss function to train the…”
On Universal Equivariant Set Networks (2019, positive): “To build the MLPs φ , ψ we will ﬁrst construct ψ to approximate q , that is, we use the universality of MLPS (see (Hornik, 1991; Sonoda & Murata, 2017; Hanin & Sellke, 2017)) to construct ψ so that ψ ( x , s 1 , . . . , s t ) − q ( x , s 1 , . . . , s t ) ∞ < 2 for all ( x , s 1 , . . . , s t ) ∈…”