A comparison of deep networks with ReLU activation function and linear spline-type methods

By Konstantin Eckle, Johannes Schmidt-Hieber. Neural Networks, 2018

370 citations recorded by the source corpus.

Deep neural networks (DNNs) generate much richer function spaces than shallow networks. Since the function spaces induced by shallow networks have several approximation theoretic drawbacks, this explains, however, not necessarily the success of deep networks. In this article we take another route by comparing the expressive power of DNNs with ReLU activation function to linear spline methods. We show that MARS (multivariate adaptive regression splines) is improper learnable by DNNs in the sense that for any given function that can be expressed as a function in MARS with M parameters there exists a multilayer neural network with O(Mlog(M∕ε)) parameters that approximates this function up to sup-norm error ε. We show a similar result for expansions with respect to the Faber-Schauder system. Based on this, we derive risk comparison inequalities that bound the statistical risk of fitting a neural network by the statistical risk of spline-based methods. This shows that deep networks perform better or only slightly worse than the considered spline methods. We provide a constructive proof for the function approximations.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/99d1bd769225bcdafb9cfa58898ca91f89305775

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

Citations in the corpus

Nonparametric regression using deep neural networks with ReLU activation function (2017, background): “We would like to refer to the simulation study in [15], which finds that for regression problems, the performance of deep neural networks is not far off from the theoretical bounds.”
The Kolmogorov-Arnold representation theorem revisited (2020, positive): “By changing the network parameters in the ﬁrst layers, the unit cube [0 , 1] d can be split in more general subsets and similar function systems as the ones underlying MARS or CART can be generated using deep ReLU networks, see also [5, 14].”
Activation functions in deep learning: A comprehensive survey and benchmark (2021, methodology): “The ReLU based deep networks perform superior or mildly worse than the spline methods [139].”