Networks and the best approximation property

By Federico Girossi, Tomaso A. Poggio. 1990

Networks can be considered as approximation schemes. Multilayer networks of the perceptron type can approximate arbitrarily well continuous functions (Cybenko 1988, 1989; Funahashi 1989; Stinchcombe and White 1989). We prove that networks derived from regularization theory and including Radial Basis Functions (Poggio and Girosi 1989), have a similar property. From the point of view of approximation theory, however, the property of approximating continuous functions arbitrarily well is not sufficient for characterizing good approximation schemes. More critical is the property ofbest approximation. The main result of this paper is that multilayer perceptron networks, of the type used in backpropagation, do not have the best approximation property. For regularization networks (in particular Radial Basis Function networks) we prove existence and uniqueness of best approximation.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Kernels for Vector-Valued Functions: a Review (2011, background): “We will first describe a regularization (frequentist) perspective (see [35, 105, 100, 86]).”
Capacity of reproducing kernel spaces in learning theory (2003, background): “If the hypothesis space is taken to be the RKHS and , then the empirical target function defined by (1.2) is in (see [9], [22], [20], [8]).”
State of the art of control schemes for smart systems featuring magneto-rheological materials (2016, background): “NNs can be also a controller or hybrid controller combined with other control algorithms for a semi-active control system [71, 71, 210, 214].”
Hierarchical learning : theory with applications in speech and vision (2009, background): “1 Deep Neural Networks/Deep Belief Networks Much work has been done to understand single (hidden) layer neural networks in terms of statistical learning theory [79, 43].”
In Defense of One-Vs-All Classification (2004, background): “In 1989, Girosi and Poggio considered regularized classification and regression problems with the square loss (Girosi and Poggio, 1989, Poggio and Girosi, 1990).”
Deep Convolutional Networks are Hierarchical Kernel Machines (2015, background): “er of (reusable) parts. Thus for problems that are compositional, such as text and images, hierarchical architectures of HBF modules minimize memory requirements. Classical theorems (see refrences in [18, 19] show that one hidden layer networks can approximate arbitrarily well rather general classes”
Why and when can deep-but not shallow-networks avoid the curse of dimensionality: A review (2016, background): “Classical theorems (see refrences in [54, 55] show that one hidden layer networks can approximate arbitrarily well rather general classes of functions.”
Face recognition with radial basis function (RBF) neural networks (2002, background): “• They possess the best approximation property [18].”
Neural Networks and Deep Learning (2018, unknown): “, Fisher discriminant) and L1-loss classification that use the binary class variables as the regression responses [139].”
On the Expressive Power of Deep Learning: A Tensor Analysis (2015, unknown): “In the next subsections we utilize tensor decompositions (factorizations) to address this computational challenge, and show how they are naturally realized by convolutional arithmetic circuits.”