Characterizing signal propagation to close the performance gap in unnormalized ResNets

By Andrew Brock, Soham De, Samuel L. Smith. 2021

145 citations recorded by the source corpus.

Batch Normalization is a key component in almost all state-of-the-art image classifiers, but it also introduces practical challenges: it breaks the independence between training examples within a batch, can incur compute and memory overhead, and often results in unexpected bugs. Building on recent theoretical analyses of deep ResNets at initialization, we propose a simple set of analysis tools to characterize signal propagation on the forward pass, and leverage these tools to design highly performant ResNets without activation normalization layers. Crucial to our success is an adapted version of the recently proposed Weight Standardization. Our analysis tools show how this technique preserves the signal in networks with ReLU or Swish activation functions by ensuring that the per-channel activation means do not grow with depth. Across a range of FLOP budgets, our networks attain performance competitive with the state-of-the-art EfficientNets on ImageNet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Optimization & Training

Citations in the corpus

High-Performance Large-Scale Image Recognition Without Normalization (2021, methodology): “The non-standard scalar multiplier of 2 is used following Brock et al. (2021) to maintain signal variance.”
Continual Normalization: Rethinking Batch Normalization for Online Continual Learning (2022, background): “Although recent efforts have shown promising results in training deep networks without BN in a single task learning (Brock et al., 2021), we argue that BN has a huge impact on continual learning.”
Going deeper with Image Transformers (2021, background): “How to normalize, weigh, or initialize the residual blocks of a residual architecture has received significant attention both for convolutional neural networks [7, 8, 28, 76] and for transformers applied to NLP or speech tasks [2, 34, 76].”
Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error (2021, methodology): “NFNets comprise a family of models, denoted by NFNet-Fx, where x ∈ { 1 , 2 , 3 , 4 , 5 , 6 } .”
How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy (2023, background): “In contrast, non-private training of the same model (NF-Resnet50 (Brock et al., 2021)) achieves 76.”
TAN without a burn: Scaling Laws of DP-SGD (2022, methodology): “Brock et al. (2021a) suggest that signal propagation improves when the order is reversed (to CONV-ReLU-GN).”
Parameter Prediction for Unseen Deep Architectures (2021, background): “Thus, many architectures are not realizable within this design space, including: VGG [25], ResNets [8], MobileNet [33] or more recent ones such as Visual Transformer (ViT) [20] and Normalization-free networks [36, 37].”
Unlocking High-Accuracy Differentially Private Image Classification through Scale (2022, methodology): “A range of other alternatives to batch normalization have recently been identiﬁed in non-private deep learning (Brock et al., 2021a; De and Smith, 2020; Kolesnikov et al., 2020; Zhang et al., 2018b).”
Analyzing and Improving the Training Dynamics of Diffusion Models (2023, unknown): “Inspired by the solutions adopted in StyleGAN2 [32] and other works that have sought alternatives to explicit normalization [1, 6, 37], we choose to modify the network so that individual layers and pathways preserve the activation magnitudes on expectation , with the goal of removing or at least…”