Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks

By Yu Bai, Jason D. Lee. 2019

130 citations recorded by the source corpus.

Recent theoretical work has established connections between over-parametrized neural networks and linearized models governed by he Neural Tangent Kernels (NTKs). NTK theory leads to concrete convergence and generalization results, yet the empirical performance of neural networks are observed to exceed their linearized models, suggesting insufficiency of this theory. Towards closing this gap, we investigate the training of over-parametrized neural networks that are beyond the NTK regime yet still governed by the Taylor expansion of the network. We bring forward the idea of \emph{randomizing} the neural networks, which allows them to escape their NTK and couple with quadratic models. We show that the optimization landscape of randomized two-layer networks are nice and amenable to escaping-saddle algorithms. We prove concrete generalization and expressivity results on these randomized networks, which lead to sample complexity bounds (of learning certain simple functions) that match the NTK and can in addition be better by a dimension factor when mild distributional assumptions are present. We demonstrate that our randomization technique can be generalized systematically beyond the quadratic case, by using it to find networks that are coupled with higher-order terms in their Taylor series.

Subfields: Artificial Intelligence, Scaling & Learning Theory, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Scaling & Learning Theory

In Statistical Learning Theory

Citations in the corpus

The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks (2020, neutral): “[2020], Bai and Lee [2020], etc.”
Dynamically Stable Infinite-Width Limits of Neural Classifiers (2020, background): “While being theoretically appealing, "laziness" assumption turns out to have a number of limitations in explaining the success of deep learning [11, 13, 14].”
TRAK: Attributing Model Behavior at Scale (2023, unknown): “For example, incorporating higher order approximations (Huang & Yau, 2020; Bai & Lee, 2020) beyond the linear approximation used in TRAK is a possible direction.”
Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK (2020, background): “Allen-Zhu et al. (2019a); Bai and Lee (2019) go beyond NTK by studying quadratic approximations of neural networks.”