Freeze and Chaos for DNNs: an NTK view of Batch Normalization, Checkerboard and Boundary Effects

By Arthur Jacot, Franck Gabriel, Clément Hongler. 2019

25 citations recorded by the source corpus.

In this paper, we analyze a number of architectural features of Deep Neural Networks (DNNs), using the so-called Neural Tangent Kernel (NTK). The NTK describes the training trajectory and generalization of DNNs in the infinite-width limit. In this limit, we show that for (fully-connected) DNNs, as the depth grows, two regimes appear:"freeze"(also known as"order"), where the (scaled) NTK converges to a constant (slowing convergence), and"chaos", where it converges to a Kronecker delta (limiting generalization). We show that when using the scaled ReLU as a nonlinearity, we naturally end up in the"freeze". We show that Batch Normalization (BN) avoids the freeze regime by reducing the importance of the constant mode in the NTK. A similar effect is obtained by normalizing the nonlinearity which moves the network to the chaotic regime. We uncover the same"freeze"and"chaos"modes in Deep Deconvolutional Networks (DC-NNs). The"freeze"regime is characterized by checkerboard patterns in the image space in addition to the constant modes in input space. Finally, we introduce a new NTK-based parametrization to eliminate border artifacts and we propose a layer-dependent learning rate to improve the convergence of DC-NNs. We illustrate our findings by training DCGANs using our setup. When trained in the"freeze"regime, we see that the generator collapses to a checkerboard mode. We also demonstrate numerically that the generator collapse can be avoided and that good quality samples can be obtained, by tuning the nonlinearity to reach the"chaos"regime (without using batch normalization).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Hardware and Architecture, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Hardware and Architecture

In Optimization & Training

Citations in the corpus

High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “…depth increases, and introduces a ‘mean-shift’ in the activations of different training examples on any single channel proportional to the network depth (De & Smith, 2020), which can cause deep networks to predict the same label for all training examples at initialization (Jacot et al., 2019).”
On the infinite width limit of neural networks with a standard parameterization (2020, background): “Infinite width Bayesian [21, 15, 17, 18, 23, 9, 7, 28, 29, 6] and gradient descent trained [12, 16, 5, 28, 13, 8, 3, 1, 2, 26] neural networks are an area of active and extremely promising work.”
Batch Normalization Biases Deep Residual Networks Towards Shallow Paths (2020, background): “However while batch normalization does improve the conditioning of a network (Jacot et al., 2019), this conditioning decays rapidly as the network depth increases if skip connections are not present (Yang et al., 2019).”
Characterizing signal propagation to close the performance gap in unnormalized ResNets (2021, background): “This indicates a large “mean shift”, whereby the hidden activations for different training inputs (in this case different vectors sampled from the unit normal) are strongly correlated (Jacot et al., 2019; Ruff et al., 2019).”
Disentangling trainability and generalization in deep learning (2019, background): “Jacot et al. (2019) demonstrate that batch normalization helps removes the “ordered phase” (as in Yang et al. (2019)) and a layer-dependent learning rate allows every layer in a netwrok to contribute to learning.”