Evolving Normalization-Activation Layers

By Hanxiao Liu, Andrew Brock, Karen Simonyan, Quoc V. Le. 2020

93 citations recorded by the source corpus.

Normalization layers and activation functions are fundamental components in deep networks and typically co-locate with each other. Here we propose to design them using an automated approach. Instead of designing them separately, we unify them into a single tensor-to-tensor computation graph, and evolve its structure starting from basic mathematical functions. Examples of such mathematical functions are addition, multiplication and statistical moments. The use of low-level mathematical functions, in contrast to the use of high-level modules in mainstream NAS, leads to a highly sparse and large search space which can be challenging for search methods. To address the challenge, we develop efficient rejection protocols to quickly filter out candidate layers that do not work well. We also use multi-objective evolution to optimize each layer's performance across many architectures to prevent overfitting. Our method leads to the discovery of EvoNorms, a set of new normalization-activation layers with novel, and sometimes surprising structures that go beyond existing design patterns. For example, some EvoNorms do not assume that normalization and activation functions must be applied sequentially, nor need to center the feature maps, nor require explicit activation functions. Our experiments show that EvoNorms work well on image classification models including ResNets, MobileNets and EfficientNets but also transfer well to Mask R-CNN with FPN/SpineNet for instance segmentation and to BigGAN for image synthesis, outperforming BatchNorm and GroupNorm based layers in many cases.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Quoc V. Le
2. Hanxiao Liu
3. Zihang Dai
4. Jianfeng Gao
5. Mohit Bansal
6. Ling Shao
7. David R. So
8. Luke S. Zettlemoyer

In Computer Vision and Pattern Recognition

1. Ling Shao
2. Luc Van Gool
3. Dacheng Tao
4. Quoc V. Le
5. Alan L. Yuille
6. Ming-Hsuan Yang
7. Xiaogang Wang
8. Trevor Darrell

In Model Architectures

1. Quoc V. Le
2. Hanxiao Liu
3. Zihang Dai
4. David R. So
5. Noam Shazeer
6. Ling Shao
7. Tie-Yan Liu
8. Wojciech Ma'nke

In Optimization & Training

1. Hanxiao Liu
2. David R. So
3. Quoc V. Le
4. Zihang Dai
5. Noam Shazeer
6. Jie Qin
7. Ling Shao
8. Fan Zhu

Citations in the corpus

• Primer: Searching for Efficient Transformers for Language Modeling (2021, methodology): “Given a search space of TensorFlow programs, we use evolution [11, 12, 13, 14, 15, 16, 17] to search for models that achieve as low of a validation loss as possible given a fixed amount of training compute.”
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “[120] searched for a combination of normalization-activation layers using AutoML [121], leading to the discovery of EvoNorms, a set of new normalization-activation layers with sometimes surprising structures that go beyond existing design patterns.”