Neural Kernels Without Tangents

By Vaishaal Shankar, Alex Fang, Wenshuo Guo, Sara Fridovich-Keil, Ludwig Schmidt, Jonathan Ragan-Kelley, B. Recht. 2020

92 citations recorded by the source corpus.

We investigate the connections between neural networks and simple building blocks in kernel space. In particular, using well established feature space tools such as direct sum, averaging, and moment lifting, we present an algebra for creating "compositional" kernels from bags of features. We show that these operations correspond to many of the building blocks of "neural tangent kernels (NTK)". Experimentally, we show that there is a correlation in test error between neural network architectures and the associated kernels. We construct a simple neural network architecture using only 3x3 convolutions, 2x2 average pooling, ReLU, and optimized with SGD and MSE loss that achieves 96% accuracy on CIFAR10, and whose corresponding compositional kernel achieves 90% accuracy. We also use our constructions to investigate the relative performance of neural networks, NTKs, and compositional kernels in the small dataset regime. In particular, we find that compositional kernels outperform NTKs and neural networks outperform both kernel methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Optimization & Training, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Greg Yang
2. Yuanzhi Li
3. Zeyuan Allen-Zhu
4. Jaehoon Lee
5. Jason D. Lee
6. Andrea Montanari
7. Lechao Xiao
8. Timothy Nguyen

In Computer Vision and Pattern Recognition

1. Zeyuan Allen-Zhu
2. Yuanzhi Li
3. Timothy Nguyen
4. Jaehoon Lee
5. Lechao Xiao
6. Roman Novak
7. Yasaman Bahri
8. Andrea Montanari

In Statistical and Nonlinear Physics

1. Yuanzhi Li
2. Greg Yang
3. Andrea Montanari
4. Zeyuan Allen-Zhu
5. Jaehoon Lee
6. Ethan Dyer
7. Lechao Xiao
8. Timothy Nguyen

In Optimization & Training

1. Zeyuan Allen-Zhu
2. Yuanzhi Li
3. Andrea Montanari
4. Song Mei
5. Behrooz Ghorbani
6. Jaehoon Lee
7. Roman Novak
8. Theodor Misiakiewicz

In Statistical Learning Theory

1. Yuanzhi Li
2. Zeyuan Allen-Zhu
3. Andrea Montanari
4. Song Mei
5. Behrooz Ghorbani
6. Theodor Misiakiewicz
7. Yasaman Bahri
8. Timothy Nguyen

Citations in the corpus

• When do neural networks outperform kernel methods? (2020, methodology): “In the right frame, we use CIFAR-10 data and compare a Myrtle-5 network (a lightweight convolutional architecture [26, 29]) with the corresponding NT KRR.”
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning (2020, methodology): “We use ZCA data preprocessing which has been reported very helpful for improving neural kernel methods’ performance together with cutout augmentation [77].”
• Explaining neural scaling laws (2021, result): “Crucially, our predictions agree with empirical findings in settings which have often proven challenging for theory – deep neural networks on real datasets.”
• Dataset Distillation with Infinitely Wide Convolutional Networks (2021, methodology): “…have been shown to achieve SOTA (among non-parametric kernels) results on image classification tasks [Novak et al., 2019, Arora et al., 2019, Li et al., 2019, Shankar et al., 2020, Bietti, 2021] and even rival finite-width networks in certain settings [Arora et al., 2020, Lee et al., 2020].”