Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefits

By Xialiang Dou, Tengyuan Liang. 2019

42 citations recorded by the source corpus.

Abstract Consider the problem: given the data pair drawn from a population with , specify a neural network model and run gradient flow on the weights over time until reaching any stationarity. How does ft , the function computed by the neural network at time t, relate to , in terms of approximation and representation? What are the provable benefits of the adaptive representation by neural networks compared to the pre-specified fixed basis representation in the classical nonparametric literature? We answer the above questions via a dynamic reproducing kernel Hilbert space (RKHS) approach indexed by the training process of neural networks. Firstly, we show that when reaching any local stationarity, gradient flow learns an adaptive RKHS representation and performs the global least-squares projection onto the adaptive RKHS, simultaneously. Secondly, we prove that as the RKHS is data-adaptive and task-specific, the residual for lies in a subspace that is potentially much smaller than the orthogonal complement of the RKHS. The result formalizes the representation and approximation benefits of neural networks. Lastly, we show that the neural network function computed by gradient flow converges to the kernel ridgeless regression with an adaptive kernel, in the limit of vanishing regularization. The adaptive kernel viewpoint provides new angles of studying the approximation, representation, generalization, and optimization advantages of neural networks.

Subfields: Artificial Intelligence, Model Architectures, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Statistical Learning Theory

Citations in the corpus

Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian (2019, background): “An interesting line of work establish connection between kernel methods and neural networks and study the generalization abilities of kernel methods when the model interpolates the training data [19, 10, 11, 12, 38, 13].”
Deep Neural Networks for Estimation and Inference: Application to Causal Effects and Other Semiparametric Estimands (2018, background): “Proofs and further discussion of similar results can be found in Farrell (2015); Chernozhukov et al.”
Just Interpolate: Kernel "Ridgeless" Regression Can Generalize (2018, background): “We leave further investigations on the connection between kernel ridgeless regression and two-layer neural networks as a future work (Dou and Liang, 2019).”
Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representations (2019, background): “Recently, a different line of papers (Mei et al., 2018; Chizat & Bach, 2018; Sirignano & Spiliopoulos, 2019; Rotskoff & Vanden-Eijnden, 2018; E, 2019b; Mei et al., 2019; Dou & Liang, 2019; Wei et al., 2018) applied the mean filed view to study the over-parameterized two level NNs.”
Geometric compression of invariant manifolds in neural nets (2020, background): “In the feature learning regime [10, 11], weights and neuron activities significantly change, the NTK evolves in time [10, 21] and compression can in principle occur.”
Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel (2018, background): “[43], Chizat and Bach [15], Sirignano and Spiliopoulos [65], Dou and Liang [18], Mei et al.”
Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit (2019, unknown): “We thus recover –from a different viewpoint– a connection with kernel methods that has been investigated in several recent papers Jacot et al. (2018); Du et al.”
Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks (2020, unknown): “…analysis for NNs (Mei et al., 2018; Chizat & Bach, 2018; Sirig-nano & Spiliopoulos, 2019a; Rotskoff & Vanden-Eijnden, 2018; Mei et al., 2019; Dou & Liang, 2019; Wei et al., 2018; Sirignano & Spiliopoulos, 2019b; Fang et al., 2019; Ara´ujo et al., 2019; Nguyen & Pham, 2020; Chen et al.,…”