Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks

By Sanjeev Arora, Simon Shaolei Du, Zhiyuan Li, Ruslan Salakhutdinov, Ruosong Wang, Dingli Yu. 2019

171 citations recorded by the source corpus.

Recent research shows that the following two models are equivalent: (a) infinitely wide neural networks (NNs) trained under l2 loss by gradient descent with infinitesimally small learning rate (b) kernel regression with respect to so-called Neural Tangent Kernels (NTKs) (Jacot et al., 2018). An efficient algorithm to compute the NTK, as well as its convolutional counterparts, appears in Arora et al. (2019a), which allowed studying performance of infinitely wide nets on datasets like CIFAR-10. However, super-quadratic running time of kernel methods makes them best suited for small-data tasks. We report results suggesting neural tangent kernels perform strongly on low-data tasks. 1. On a standard testbed of classification/regression tasks from the UCI database, NTK SVM beats the previous gold standard, Random Forests (RF), and also the corresponding finite nets. 2. On CIFAR-10 with 10 - 640 training samples, Convolutional NTK consistently beats ResNet-34 by 1% - 3%. 3. On VOC07 testbed for few-shot image classification tasks on ImageNet with transfer learning (Goyal et al., 2019), replacing the linear SVM currently used with a Convolutional NTK SVM consistently improves performance. 4. Comparing the performance of NTK with the finite-width net it was derived from, NTK behavior starts at lower net widths than suggested by theoretical analysis(Arora et al., 2019a). NTK's efficacy may trace to lower variance of output.

Subfields: Scaling & Learning Theory, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Scaling & Learning Theory

In Statistical Learning Theory

Citations in the corpus

When do neural networks outperform kernel methods? (2020, background): “Some empirical studies on various datasets showed that networks can be replaced by suitable kernels with limited drop in performances [5, 21, 20, 24, 19, 10, 14, 29].”
Differentially Private Learning Needs Better Features (or Much More Data) (2020, background): “Recent work on Neural Tangent Kernels (Jacot et al., 2018) has shown that the performance of deep neural networks on CIFAR-10 could be matched by specialized kernel methods (Li et al., 2019; Arora et al., 2020; Shankar et al., 2020).”
Few-Shot Learning via Learning the Representation, Provably (2020, methodology): “In particular, representation learning is widely adopted for few-shot learning tasks (Sun et al., 2017, Goyal et al., 2019, Arora et al., 2020).”
Neural Tangents: Fast and Easy Infinite Neural Networks in Python (2019, background): “Various prior works have evaluated convolutional and fully-connected models on certain datasets (Lee et al., 2018; Matthews et al., 2018a,b; Novak et al., 2019; Garriga-Alonso et al., 2019; Arora et al., 2019).”
Deep Learning on a Data Diet: Finding Important Examples Early in Training (2021, background): “However, finite neural networks outperform their infinitewidth limits [17] and have different dynamics early in training [18].”
: "Statistical Modeling: The Two Cultures" (Statistical Science, 2001, 16(3), 199-231) (unknown): “Paradoxically, Tikhonov regularization (Tikhonov, 1943) characterizes much of the early nonparametrics works (Hoerl and Kennard, 1970), and it also characterizes current theoretical underpinnings of deep neural networks; e.g., Jacot et al. (2018); Arora et al. (2019).”
Dataset Distillation with Infinitely Wide Convolutional Networks (2021, background): “…have been shown to achieve SOTA (among non-parametric kernels) results on image classification tasks [Novak et al., 2019, Arora et al., 2019, Li et al., 2019, Shankar et al., 2020, Bietti, 2021] and even rival finite-width networks in certain settings [Arora et al., 2020, Lee et al., 2020].”
On the Similarity between the Laplace and Neural Tangent Kernels (2020, methodology): “NTK Specifications We reproduced the results of [4] using the publicly available code1, and followed the same protocol as in [4].”
Dataset Meta-Learning from Kernel Ridge-Regression (2020, methodology): “2560 13.0±0.5 39.1±0.7 40.2±0.7 - - 5000 17.8±0.4 42.1±0.5 43.7±0.6 - - 10000 24.9±0.6 45.3±0.6 47.7±0.6 - - † Conv14 kernel with global average pooling (Arora et al., 2019b) ‡ Myrtle10-Gaussian kernel (Shankar et al., 2020) E.2 KIP AND LS TRANSFER ACROSS KERNELS Figure A1 plots how KIP (with only…”
How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks (2020, methodology): “Our code for exact computation of NTK is adapted from Arora et al. (2020); Novak et al. (2020).”
Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks (2021, methodology): “UCI Dataset We applied ResNTK and FC-NTK to 90 datasets of the UCI collection (< 5000 items) using the protocol of (Arora et al., 2020).”
On the infinite width limit of neural networks with a standard parameterization (2020, background): “Infinite width Bayesian [21, 15, 17, 18, 23, 9, 7, 28, 29, 6] and gradient descent trained [12, 16, 5, 28, 13, 8, 3, 1, 2, 26] neural networks are an area of active and extremely promising work.”
Neural Kernels Without Tangents (2020, methodology): “For experiments on the UCI datasets, we minimize the hinge loss with libSVM to appropriately compare with prior work (Arora et al., 2020; Fernández-Delgado et al., 2014).”
Tensor Programs II: Neural Tangent Kernel for Any Architecture (2020, result): “Compared to neural networks, kernel regression with the corresponding NTKs work better in the low data regime [7], consistent with classical observations about kernel methods and previous works on NNGPs [44, 52].”