Fast and Accurate Model Scaling

By Piotr Doll'ar, Mannat Singh, Ross B. Girshick. 2021

113 citations recorded by the source corpus.

In this work we analyze strategies for convolutional neural network scaling; that is, the process of scaling a base convolutional network to endow it with greater computational complexity and consequently representational power. Example scaling strategies may include increasing model width, depth, resolution, etc. While various scaling strategies exist, their tradeoffs are not fully understood. Existing analysis typically focuses on the interplay of accuracy and flops (floating point operations). Yet, as we demonstrate, various scaling strategies affect model parameters, activations, and consequently actual runtime quite differently. In our experiments we show the surprising result that numerous scaling strategies yield networks with similar accuracy but with widely varying properties. This leads us to propose a simple fast compound scaling strategy that encourages primarily scaling model width, while scaling depth and resolution to a lesser extent. Unlike currently popular scaling strategies, which result in about O(s) increase in model activation w.r.t. scaling flops by a factor of s, the proposed fast compound scaling results in close to $O\left( {\sqrt s } \right)$ increase in activations, while achieving excellent accuracy. Fewer activations leads to speedups on modern memory-bandwidth limited hardware (e.g., GPUs). More generally, we hope this work provides a framework for analyzing scaling strategies under various computational constraints.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Early Convolutions Help Transformers See Better (2021, methodology): “We use the best optimizer choice for each model (AdamW for ViT models and SGD for RegNetY models).”
ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training (2021, result): “This concurs with recent studies [2, 12] that better disentangle the benefits from the architectures from those of the training scheme.”
ResNet strikes back: An improved training procedure in timm (2021, methodology): “[10] Piotr Dollár, Mannat Singh, and Ross B.”
Performance-Efficiency Trade-Offs in Unsupervised Pre-Training for Speech Recognition (2021, methodology): “Scaling Up We adopt a simple scaling up recipe, leaving finding more optimal scaled-up configurations, an open research problem (Tan and Le, 2019; Dollár et al., 2021), for future work.”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, unknown): “In the HiP-MDP setting, they utilized the idea of a common structure and extended to enable state abstraction inspired by block MDP. Dollar et al. [324] proposed a simple and fast complex scaling strategy that scales the underlying convolutional network to give greater computational complexity and,…”
MViTv2: Improved Multiscale Vision Transformers for Classification and Detection (2021, background): “CNNs serve as the primary backbones for computer vision tasks, including image recognition [10,15,34,39,46,48,62, 67,69,71], object detection [6,29,38,53,63,93] and video recognition [8,22,23,25,28,42,51,61,66,75,79,84,92].”
YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors (2022, methodology): “In [15], the researcher analyzes the relationship between scaling factors and the amount of parameters and operations, trying to directly estimate some rules, and thereby obtain the scaling factors required by model scaling.”
MPViT: Multi-Path Vision Transformer for Dense Prediction (2021, background): “Moreover, to strike the balanced tradeoff between accuracy/speed, we will further consider path dimensions in a compound scaling strategy [15,49], to consider the optimal combination of depths, widths, and resolutions when increasing/decreasing the model capacity.”
You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection (2021, methodology): “From the model scaling perspective [20, 56, 61], the small and base models of YOLOS / DeiT can be seen as performing width scaling ( w ) [30, 68] on the corresponding tiny model.”
Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs (2022, background): “For CNNs, existing scaling approaches usually focus on model depth, width, input resolution [32, 70, 84], bottleneck ratio and group width [32, 70].”