High-Performance Large-Scale Image Recognition Without Normalization

By Andrew Brock, Soham De, Samuel L. Smith, Karen Simonyan. 2021

638 citations recorded by the source corpus.

Batch normalization is a key component of most image classification models, but it has many undesirable properties stemming from its dependence on the batch size and interactions between examples. Although recent work has succeeded in training deep ResNets without normalization layers, these models do not match the test accuracies of the best batch-normalized networks, and are often unstable for large learning rates or strong data augmentations. In this work, we develop an adaptive gradient clipping technique which overcomes these instabilities, and design a significantly improved class of Normalizer-Free ResNets. Our smaller models match the test accuracy of an EfficientNet-B7 on ImageNet while being up to 8.7x faster to train, and our largest models attain a new state-of-the-art top-1 accuracy of 86.5%. In addition, Normalizer-Free models attain significantly better performance than their batch-normalized counterparts when finetuning on ImageNet after large-scale pre-training on a dataset of 300 million labeled images, with our best models obtaining an accuracy of 89.2%. Our code is available at https://github.com/deepmind/ deepmind-research/tree/master/nfnets

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

VOLO: Vision Outlooker for Visual Recognition (2021, methodology): “2 NFNet-F5 [2] CNN 377M 290B 416 544 86.”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, positive): “We observed only marginal improvements in performance from this change, but it brings the augmentation strategy more in line with the strategy used elsewhere in the literature (Brock et al., 2021; Touvron et al., 2021a).”
Perceiver: General Perception with Iterative Attention (2021, background): “Note that state-of-the art on ImageNet without pretraining was 86.5% top-1 accuracy at submission (Brock et al., 2021).”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, methodology): “Our vision encoder is a pretrained and frozen NormalizerFree ResNet (NFNet) [10] – we use the F6 model.”
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, background): “Adaptation without updating the layer normalization can be a promising direction to reduce the parameters further, for instance by building on recent advances in normalization-free models [35], which we leave to future work.”
Stable and low-precision training for large-scale vision-language models (2023, background): “Consequently, researchers have proposed initialization and scaling schemes which remedy this issue [1, 70, 4, 18].”
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, methodology): “Frozen [49] achieves this by jointly training an NF-ResNet-50 [3] and frozen GPT-3 with the Conceptual Captioning dataset [44].”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “Frozen afﬁxes an NFResnet [5] vision encoder to a pretrained autoregressive LM and, keeping the LM weights frozen, trains the vision encoder along with a visual preﬁx that linearly maps the output of the vision encoder to the dimensionality of the LM’s token embeddings.”
Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “More recent image encoders can overcome such inexact computations because they avoid batch normalization by replacing it with layer normalization like Vision Transformer (Dosovitskiy et al., 2021), or just do not use any normalization at all like NFNet (Brock et al., 2021).”
Multimodal Few-Shot Learning with Frozen Language Models (2021, methodology): “Vision Encoder Our vision encoder is based on NF-ResNet-50 [3].”
CoAtNet: Marrying Convolution and Attention for All Data Sizes (2021, methodology): “Subsequent works use special regularization and stronger data augmentation to improve the vanilla ViT [16, 17, 18], yet none of these ViT variants could outperform the SOTA convolution-only models on ImageNet classification given the same amount of data and computation [19, 20].”
MLP-Mixer: An all-MLP Architecture for Vision (2021, methodology): “NFNets [7] are normalizer-free ResNets with several optimizations for ImageNet classification.”
Going deeper with Image Transformers (2021, background): “It is the state of the art, on par with a recent concurrent work [8] that has a significantly higher number of FLOPs.”
EfficientNetV2: Smaller Models and Faster Training (2021, methodology): “…al., 2021) 84.7% 133M 36B 70 20 NFNet-F2 (Brock et al., 2021) 85.1% 194M 63B 124 36 NFNet-F3 (Brock et al., 2021) 85.7% 255M 115B 203 65 NFNet-F4 (Brock et al., 2021) 85.9% 316M 215B 309 126 LambdaResNet-420-hybrid (Bello, 2021) 84.9% 125M - - 67 BotNet-T7-hybrid (Srinivas et al., 2021) 84.7%…”
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference (2021, background): “Since our approach specializes in the high-throughput regime, we do not include very large and slow models [61, 62].”
XCiT: Cross-Covariance Image Transformers (2021, methodology): “We compare to the state-of-the-art convolutional and transformer-based architectures [10, 44, 53, 58, 67] in Table 2.”
Revisiting ResNets: Improved Training and Scaling Strategies (2021, methodology): “We note that similar scaling strategies are also employed in recent works that obtain large speed-ups over EfficientNets such as LambdaResNets (Bello, 2021) and NFNets (Brock et al., 2021).”
Analyzing and Improving the Training Dynamics of Diffusion Models (2023, positive): “By now we can expect these to be standardized, and thus the balance between the branches is exposed as a meaningfully controllable parameter [7].”
GemNet: Universal Directional Graph Neural Networks for Molecules (2021, background): “We found that other measures such as adaptive gradient clipping [7], scaled weight standardization [6], or weighting the residual block with zero at initialization [16] are not beneficial for model accuracy.”