Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics
By Greg Yang, Etai Littwin. 2021
81 citations recorded by the source corpus.
Yang (2020a) recently showed that the Neural Tangent Kernel (NTK) at initialization has an infinite-width limit for a large class of architectures including modern staples such as ResNet and Transformers. However, their analysis does not apply to training. Here, we show the same neural networks (in the so-called NTK parametrization) during training follow a kernel gradient descent dynamics in function space, where the kernel is the infinite-width NTK. This completes the proof of the *architectural universality* of NTK behavior. To achieve this result, we apply the Tensor Programs technique: Write the entire SGD dynamics inside a Tensor Program and analyze it via the Master Theorem. To facilitate this proof, we develop a graphical notation for Tensor Programs.
Subfields: Artificial Intelligence, Computational Mathematics, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Shotaro Akaho
- Ryo Karakida
- Greg Yang
- Karen Livescu
- Dacheng Tao
- Weiran Wang
- Indrė Žliobaitė
- Xintao Wu
In Computer Vision and Pattern Recognition
- Shotaro Akaho
- Ryo Karakida
- Karen Livescu
- Dacheng Tao
- Weiran Wang
- Kristen Grauman
- Lior Wolf
- Changxing Ding
Citations in the corpus
- Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting (2021, background): “Studies on NTK eigenvalues have made remarkable progress, covering shallow and deep ReLU neural networks (Geifman et al., 2020; Chen & Xu, 2020), skip connections (Belfer et al., 2021), and CNNs (Favero et al., 2021).”