Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality

By Taiji Suzuki. International Conference on Learning Representations, 2018

300 citations recorded by the source corpus.

Deep learning has shown high performances in various types of tasks from visual recognition to natural language processing, which indicates superior flexibility and adaptivity of deep learning. To understand this phenomenon theoretically, we develop a new approximation and estimation error analysis of deep learning with the ReLU activation for functions in a Besov space and its variant with mixed smoothness. The Besov space is a considerably general function space including the Holder space and Sobolev space, and especially can capture spatial inhomogeneity of smoothness. Through the analysis in the Besov space, it is shown that deep learning can achieve the minimax optimal rate and outperform any non-adaptive (linear) estimator such as kernel ridge regression, which shows that deep learning has higher adaptivity to the spatial inhomogeneity of the target function than other estimators such as linear ones. In addition to this, it is shown that deep learning can avoid the curse of dimensionality if the target function is in a mixed smooth Besov space. We also show that the dependency of the convergence rate on the dimensionality is tight due to its minimax optimality. These results support high adaptivity of deep learning and its superior ability as a feature extractor.

Subfields: Computer Vision and Pattern Recognition, Geophysics

View Paper: https://www.semanticscholar.org/paper/9f56bda06700297fe8a8aab4bb429451cee9a441

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Zhaoran Wang
2. Zhuoran Yang
3. Yuchen Xie
4. Haizhao Yang
5. G. Em Karniadakis
6. Taiji Suzuki
7. Zhongqiang Zhang
8. Zuowei Shen

In Geophysics

1. Zhuoran Yang
2. Yuchen Xie
3. Zhaoran Wang
4. G. Em Karniadakis
5. Zuowei Shen
6. Bin Han

Citations in the corpus

• Variational Physics-Informed Neural Networks For Solving Partial Differential Equations (2019, background): “There have been several works in the literature [21, 8, 25, 34, 14, 33], which attempt to identify these di ff erent sources of error and develop a proper framework for error analysis of deep NN. Achieving an optimal approximation, however, depends on many factors and it still remains an open…”
• A Theoretical Analysis of Deep Q-Learning (2019, methodology): “ion of neural networks and Dziugaite and Roy (2017); Neyshabur et al. (2018) establish the PAC-Bayes bounds for neural networks. Among these works, our work is more related to Schmidt-Hieber (2020+); Suzuki (2019), which relate the VC dimension of the ReLU networks to a set of hyperparameters used t”
• Deep Network Approximation for Smooth Functions (2020, background): “Most existing approximation theories for deep neural networks so far focus on the approximation error in the number of parameters W [4, 5, 9, 11, 12, 14, 15, 19–21, 24, 29– 33, 35–38, 43–47].”
• Generalization of Two-layer Neural Networks: An Asymptotic Viewpoint (2020, background): “Suzuki (2018); Allen-Zhu and Li (2019); Yehudai and Shamir (2019) demonstrated that neural network outperforms linear estimators (including kernel methods) in learning various target functions.”