Nonparametric regression using deep neural networks with ReLU activation function

By Johannes Schmidt-Hieber. 2017

1,061 citations recorded by the source corpus.

Consider the multivariate nonparametric regression model. It is shown that estimators based on sparsely connected deep neural networks with ReLU activation function and properly chosen network architecture achieve the minimax rates of convergence (up to $\log n$-factors) under a general composition assumption on the regression function. The framework includes many well-studied structural constraints such as (generalized) additive models. While there is a lot of flexibility in the network architecture, the tuning parameter is the sparsity of the network. Specifically, we consider large networks with number of potential network parameters exceeding the sample size. The analysis gives some insights into why multilayer feedforward neural networks perform well in practice. Interestingly, for ReLU activation function the depth (number of layers) of the neural network architectures plays an important role and our theory suggests that for nonparametric regression, scaling the network depth with the sample size is natural. It is also shown that under the composition assumption wavelet estimators can only achieve suboptimal rates.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Deep Neural Networks for Estimation and Inference: Application to Causal Effects and Other Semiparametric Estimands (2018, methodology): “However, for ζ > 1 a network with weights bounded by one (as in Schmidt-Hieber (2019)) must have width 2ζ, so ζ must be known, and yields expected risk of order ζ/n. Turning to special cases, we first show that the optimal rate of Stone (1982) can be attained, up to log factors.”
Posterior Concentration for Sparse Deep Learning (2018, methodology): “Unlike related statistical developments (Schmidt-Hieber (2017), Bauer and Kohler (2017)), we approach the reconstruction problem from a purely Bayesian point of view.”
Neural network approximation (2020, methodology): “This approach is studied, for example, in (Mhaskar & Poggio 2019, Shen et al. 2019, Schmidt-Hieber 2020).”
The Kolmogorov-Arnold representation theorem revisited (2020, background): “Approximation properties of deep neural networks for composed functions are studied in [12, 25, 21, 13, 26, 3, 23, 27, 6].”
Optimal approximation of continuous functions by very deep ReLU networks (2018, methodology): “c) This statement can be obtained by combining arguments of Theorem 4 of Yarotsky (2017) with the recently established tight upper bound for the VC dimension of ReLU networks (Bartlett et al.”
A Theoretical Analysis of Deep Q-Learning (2019, methodology): “3, obtained from Schmidt-Hieber (2017), to construct a ReLU network that approximates each hjk, which, combined with (4.”
Spectral-Pruning: Compressing deep neural network via spectral analysis (2018, background): “Fast learning rate has been also studied in deep leaning, e.g., [28, 5, 36, 6, 26, 27, 42, 25, 46], but they are not about model compression.”
A comparison of deep networks with ReLU activation function and linear spline-type methods (2018, methodology): “Lemma 4.3 (Lemma 5 in Schmidt-Hieber (2017)).”
Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality (2018, methodology): “This is almost direct consequence of Lemma 10 of Schmidt-Hieber (2018)5. The only difference is the assumption of ‖f‖∞ ≤ F for f ∈ F and f = f while Lemma 10 of Schmidt-Hieber (2018) assumed 0 ≤ f(x) ≤ F ′ for F ′ > 1. However, this can be easily fixed by shifting the function value by +F then the r”
Non-vacuous Generalization Bounds at the ImageNet Scale: a PAC-Bayesian Compression Approach (2018, background): “02; Langford & Caruana, 2002; Hinton & van Camp, 1993; Hochreiter & Schmidhuber, 1997; Baldassi et al., 2015; 2016; Chaudhari et al., 2017; Keskar et al., 2017; Dziugaite & Roy, 2017; Schmidt-Hieber, 2017; Neyshabur et al., 3 Published as a conference paper at ICLR 2019 2017; 2018; A”
KAN: Kolmogorov-Arnold Networks (2024, positive): “For higher dimensions, we characterize the generalization behavior of KANs with a theorem below.”
Double Debiased Machine Learning Nonparametric Inference with Continuous Treatments (2019, neutral): “…and series estimators, as well as recent ML estimators, such as Lasso (Bickel, Ritov, and Tsybakov 2009), neural networks (Chen and White 1999; Schmidt-Hieber 2020; Farrell, Liang, and Misra 2021b), random forests (Syrgkanis and Zampetakis 2020), and empirical L 2 rate for boosting (Luo and…”