Enhanced Convolutional Neural Tangent Kernels

By Zhiyuan Li, Ruosong Wang, Dingli Yu, Simon Shaolei Du, Wei Hu, Ruslan Salakhutdinov, Sanjeev Arora. 2019

138 citations recorded by the source corpus.

Recent research shows that for training with $\ell_2$ loss, convolutional neural networks (CNNs) whose width (number of channels in convolutional layers) goes to infinity correspond to regression with respect to the CNN Gaussian Process kernel (CNN-GP) if only the last layer is trained, and correspond to regression with respect to the Convolutional Neural Tangent Kernel (CNTK) if all layers are trained. An exact algorithm to compute CNTK (Arora et al., 2019) yielded the finding that classification accuracy of CNTK on CIFAR-10 is within 6-7% of that of that of the corresponding CNN architecture (best figure being around 78%) which is interesting performance for a fixed kernel. Here we show how to significantly enhance the performance of these kernels using two ideas. (1) Modifying the kernel using a new operation called Local Average Pooling (LAP) which preserves efficient computability of the kernel and inherits the spirit of standard data augmentation using pixel shifts. Earlier papers were unable to incorporate naive data augmentation because of the quadratic training cost of kernel regression. This idea is inspired by Global Average Pooling (GAP), which we show for CNN-GP and CNTK is equivalent to full translation data augmentation. (2) Representing the input image using a pre-processing technique proposed by Coates et al. (2011), which uses a single convolutional layer composed of random image patches. On CIFAR-10, the resulting kernel, CNN-GP with LAP and horizontal flip data augmentation, achieves 89% accuracy, matching the performance of AlexNet (Krizhevsky et al., 2012). Note that this is the best such result we know of for a classifier that is not a trained neural network. Similar improvements are obtained for Fashion-MNIST.

Subfields: Computer Vision and Pattern Recognition, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical Learning Theory

Citations in the corpus

When do neural networks outperform kernel methods? (2020, background): “RKHS behave well on certain image classification tasks [4, 21, 24], and this seems to contradict the previous point.”
Mixed-Privacy Forgetting in Deep Networks (2020, methodology): “Using a first-order Taylor expansion (linearization) of the network to study its behavior has gained interest recently in the NTK theory [25, 30] as a tool to study the dynamics of DNNs in the limit of infinite filters.”
Provably efficient machine learning for quantum many-body problems (2021, methodology): “Incorporating these restrictions can be achieved by using a suitable k ( x , x l ), such as one that corresponds to a large-width convolutional neural network (CNN) ( 48 ) or a graph neural network ( 49 ).”
Backward Feature Correction: How Deep Learning Performs Deep Learning (2020, background): “Figure 3: Comparison between ReLU networks, quadratic networks, and several optimized kernel methods (* for [68] and ** for [55]).”
On the Similarity between the Laplace and Neural Tangent Kernels (2020, background): “Convolutional NTKs (CNTK) were shown to express the limit of convolutional neural networks when the number of channels tends to infinity, and recent empirical results showed that the two achieve similar accuracy on test data [2, 31].”
Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK (2020, background): “…and Gu (2019); Du et al. (2018c); Arora et al. (2019a); Allen-Zhu and Li (2019b); Allen-Zhu et al. (2019c,b); Li and Liang (2018); Zou et al. (2018); Du et al. (2018a); Daniely et al. (2016); Ghorbani et al. (2019); Li et al. (2019a); Hanin and Nica (2019); Yang (2019) and the references therein.”
Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations (2020, unknown): “[3,24] extend the framework to convolutional networks.”
LESS: Selecting Influential Data for Targeted Instruction Tuning (2024, background): “Past works that instantiate this kernel have focused on vision tasks with relatively few classes (Arora et al., 2019; Yu et al., 2020a; Wei et al., 2022a) or prompt-based text classification (Malladi et al., 2023), which only considers a label word for each class in simple downstream tasks.”
Neural Kernels Without Tangents (2020, methodology): “Both of these equivalences extend to convolutional neural networks (CNNs) (Li et al., 2019; Novak et al., 2019).”