Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs

By Jonathan Frankle, David J. Schwab, Ari S. Morcos. 2020

164 citations recorded by the source corpus.

Batch normalization (BatchNorm) has become an indispensable tool for training deep neural networks, yet it is still poorly understood. Although previous work has typically focused on its normalization component, BatchNorm also adds two per-feature trainable parameters - a coefficient and a bias - whose role and expressive power remain unclear. To study this question, we investigate the performance achieved when training only these parameters and freezing all others at their random initializations. We find that doing so leads to surprisingly high performance. For example, sufficiently deep ResNets reach 82% (CIFAR-10) and 32% (ImageNet, top-5) accuracy in this configuration, far higher than when training an equivalent number of randomly chosen parameters from elsewhere in the network. BatchNorm achieves this performance in part by naturally learning to disable around a third of the random features. Not only do these results highlight the under-appreciated role of the affine parameters in BatchNorm, but - in a broader sense - they characterize the expressive power of neural networks constructed simply by shifting and rescaling random features.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Optimization & Training

Citations in the corpus

BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (2021, background): “Michel and Neubig (2018) ﬁne-tuned the biases of the output softmax in an NMT systems, to personalize the output vocabulary, and Frankle et al. (2020) have demonstrated that randomly-initialized CNNs achieve reasonable accuracy after training the batch-norm layers alone.”
Pretrained Transformers as Universal Computation Engines (2021, neutral): “Similar to a study of random CNNs by (Frankle, Schwab, and Morcos 2020), we generally find the layer norm parameters to be most important.”
Supermasks in Superposition (2020, methodology): “[9] Jonathan Frankle, David J Schwab, and Ari S Morcos.”
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little (2021, background): “, 2018) or pruning (Frankle et al., 2020), and that even randomly wired networks can perform well at image recognition (Xie et al.”
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following (2023, methodology): “In detail, the fine-tuning techniques include zero-initialized gating [20, 91], LoRA [29], and bias-norm tuning [18, 21, 81,88].”
VeRA: Vector-based Random Matrix Adaptation (2023, background): “Other works (Lu et al., 2022; Schrimpf et al., 2021; Frankle et al., 2021) have shown that frozen, randomly initialized models, with small sections finetuned, can perform surprisingly well.”
On-Device Training Under 256KB Memory (2022, background): “Later studies investigate other tuning methods including updating biases [13, 71], updating normalization layer parameters [54, 26], updating small parallel branches [13, 33], etc.”
Efficient Self-supervised Vision Transformers for Representation Learning (2021, background): “ViT-BN is ViT that has BatchNorm [21], and “/P ” denotes a patch size of P×P .”
TinyTL: Reduce Memory, Not Parameters for Efficient On-Device Learning (2020, background): “Recently, [45,46] propose to only update parameters of the batch normalization (BN) [47] layers, which greatly reduces the number of trainable parameters.”
What shapes feature representations? Exploring datasets, architectures, and training (2020, background): “Further work has shown that minimal training – only adjusting BatchNorm parameters – of untrained models already yields surprisingly high classification performance [8].”
Neural networks with late-phase weights (2020, background): “…additional parameters have large expressive power: it has been shown that learning only γ and β keeping the remaining weights frozen can lead to significantly lower loss than when learning random subsets of other weights of matching dimensionality (Frankle et al., 2020; Mudrakarta et al., 2019).”
Improving robustness against common corruptions by covariate shift adaptation (2020, background): “The expressive power of adapting only affine BN parameters BN parameters was shown in multi-task [54] and DA contexts [4] and holds even for fine-tuning randomly initialized ResNets [55].”