Designing Network Design Spaces

By Ilija Radosavovic, Raj Prateek Kosaraju, Ross B. Girshick, Kaiming He, Piotr Doll'ar. 2020

2,189 citations recorded by the source corpus.

In this work, we present a new network design paradigm. Our goal is to help advance the understanding of network design and discover design principles that generalize across settings. Instead of focusing on designing individual network instances, we design network design spaces that parametrize populations of networks. The overall process is analogous to classic manual design of networks, but elevated to the design space level. Using our methodology we explore the structure aspect of network design and arrive at a low-dimensional design space consisting of simple, regular networks that we call RegNet. The core insight of the RegNet parametrization is surprisingly simple: widths and depths of good networks can be explained by a quantized linear function. We analyze the RegNet design space and arrive at interesting findings that do not match the current practice of network design. The RegNet design space provides simple and fast networks that work well across a wide range of flop regimes. Under comparable training settings and flops, the RegNet models outperform the popular EfficientNet models while being up to 5x faster on GPUs.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Materials Chemistry, Molecular Biology, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Materials Chemistry

In Molecular Biology

In Model Architectures

Citations in the corpus

Early Convolutions Help Transformers See Better (2021, methodology): “We use the best optimizer choice for each model (AdamW for ViT models and SGD for RegNetY models).”
Emerging Properties in Self-Supervised Vision Transformers (2021, methodology): “The second set of models are trained with hard knowledge distillation from a pre-trained supervised RegNetY [56].”
Self-supervised Pretraining of Visual Features in the Wild (2021, methodology): “Of particular interest, the RegNets [40] achieve competitive performance on standard image benchmarks while offering an efﬁcient runtime and memory usage making them a candidate for training at scale.”
Training data-efficient image transformers & distillation through attention (2020, methodology): “In all of our subsequent distillation experiments the default teacher is a RegNetY-16GF [37] (84M parameters) that we trained with the same data augmentation as DeiT.”
Grafit: Learning fine-grained image representations with coarse labels (2020, methodology): “0GF [43] with resolution 384× 384 pixels that is 4× faster than EfficientNetB7 at inference.”
Scalable Diffusion Models with Transformers (2022, background): “In general, parameter counts are poor proxies for the complexity of image models since they do not account for, e.g., image resolution which signiﬁcantly impacts performance [44, 45].”
High-Performance Large-Scale Image Recognition Without Normalization (2021, methodology): “Employing our depth patterns described above, we considered a range of alternative patterns (taking inspiration from Radosavovic et al. (2020)) but found that only one choice was better than this default: [256, 512, 1536, 1536].”
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, methodology): “RegNet [44], the Swin Transformer achieves a slightly better speed-accuracy trade-off.”
The Efficiency Misnomer (2021, background): “These differences are not surprising, given that we are comparing transformer-based (Dosovitskiy et al., 2020; Liu et al., 2021) and convolution-based architectures (Tan and Le, 2019; Radosavovic et al., 2020).”
Learning to Learn with Generative Models of Neural Network Checkpoints (2022, methodology): “Past works have constructed datasets of neural networks and used them in various settings: analyzing population-level trends (Radosavovic et al., 2019; 2020), benchmarking neural architecture search (Ying et al., 2019), training hypernetworks (Knyazev et al., 2021), predicting model properties…”
VEGA: Towards an End-to-End Configurable AutoML Pipeline (2020, methodology): “2 × faster than RegNetX-32GF (Radosavovic et al., 2020) on ImageNet.”
Transformers in Vision: A Survey (2021, methodology): “Besides using augmentation and regularization procedures common in CNNs, the main contribution of DeiT [198] is a novel native distillation approach for Transformers which uses a CNN as a teacher model (RegNetY-16GF [165]) to train the Transformer model.”
Knowledge Distillation: A Survey (2020, background): “Inspired by the success of neural architecture search (or NAS), the performances of small neural networks have been further improved by searching for a global structure based on efficient meta operations or blocks (Wu et al., 2019; Tan et al., 2019; Tan and Le, 2019; Radosavovic et al., 2020).”
Going deeper with Image Transformers (2021, methodology): “We compare CaiT with DeiT [64], Vit-B [19], TNT [26], T2T [75] and to several state-of-the-art convnets: Regnet [51] improved by Touvron et al.”
Multiscale Vision Transformers (2021, methodology): “Incorporating downsampling, shift invariance, and shared weights, ConvNets are de-facto standard backbones for computer vision tasks for image [65, 62, 88, 90, 46, 12, 15, 34, 93, 81, 41] and video [87, 31, 11, 79, 69, 106, 96, 30, 105, 35, 29, 117, 57].”
ResNeSt: Split-Attention Networks (2020, methodology): “RegNet [44] conducts a systematic search for the featuremap groups in ResNet variants and achieves superior performance over EfficientNet [54].”