On Network Design Spaces for Visual Recognition

By Ilija Radosavovic, Justin Johnson, Saining Xie, Wan-Yen Lo, Piotr Doll'ar. 2019

150 citations recorded by the source corpus.

Over the past several years progress in designing better neural network architectures for visual recognition has been substantial. To help sustain this rate of progress, in this work we propose to reexamine the methodology for comparing network architectures. In particular, we introduce a new comparison paradigm of distribution estimates, in which network design spaces are compared by applying statistical techniques to populations of sampled models, while controlling for confounding factors like network complexity. Compared to current methodologies of comparing point and curve estimates of model families, distribution estimates paint a more complete picture of the entire design landscape. As a case study, we examine design spaces used in neural architecture search (NAS). We find significant statistical differences between recent NAS design space variants that have been largely overlooked. Furthermore, our analysis reveals that the design spaces for standard model families like ResNeXt can be comparable to the more complex ones used in recent NAS work. We hope these insights into distribution analysis will enable more robust progress toward discovering better networks for visual recognition.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Early Convolutions Help Transformers See Better (2021, methodology): “To quantify this, we make use of error distribution functions (EDFs) [30].”
Scalable Diffusion Models with Transformers (2022, background): “In general, parameter counts are poor proxies for the complexity of image models since they do not account for, e.g., image resolution which signiﬁcantly impacts performance [44, 45].”
Learning to Learn with Generative Models of Neural Network Checkpoints (2022, methodology): “Past works have constructed datasets of neural networks and used them in various settings: analyzing population-level trends (Radosavovic et al., 2019; 2020), benchmarking neural architecture search (Ying et al., 2019), training hypernetworks (Knyazev et al., 2021), predicting model properties…”
Designing Network Design Spaces (2020, methodology): “Our basic training settings follow [21] as discussed in §3.”
Are Labels Necessary for Neural Architecture Search? (2020, methodology): “For the random experiment analysis, we follow the setup proposed in [2] and recently adopted in [25].”
Evaluating the Search Phase of Neural Architecture Search (2019, background): “This phenomenon was already noticed by Zela et al. (2018), and concurrently to our work by Li & Talwalkar (2019); Xie et al. (2019); Ying et al. (2019), but the analysis of its impact or its causes went no further.”
A ConvNet for the 2020s (2022, background): “Notably, researchers have thoroughly investigated the distribution of computation [53, 54], and a more optimal design is likely to exist.”
Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution (2020, methodology): “Most of these methods are tailored for 2D visual recognition, which has many well-defined search spaces [44].”
Model complexity of deep learning: a survey (2021, background): “[88] investigate the network design spaces of model selection and design approaches.”
Prune and Replace NAS (2019, background): “This design space change significantly simplifies the problem, thus speeds up search, but also implicitly improves the final evaluation performance [25].”
Tent: Fully Test-Time Adaptation by Entropy Minimization (2021, methodology): “Our implementation is in PyTorch (Paszke et al., 2019) with the pycls library (Radosavovic et al., 2019).”
Efficient Architecture Search for Continual Learning (2020, unknown): “Apart from algorithms, NAS also has many valuable applications. such as image classiﬁcation [30], [31], video segmentation [32], [33], and text representation [34].”
SP-NAS: Serial-to-Parallel Backbone Search for Object Detection (2020, background): “Recently, Neural Architecture Search (NAS) algorithms demonstrate promising results on discovering top-accuracy architectures for image classiﬁcation, which surpasses the performance of hand-crafted networks and saves human’s efforts [49, 23, 45, 25, 31, 40, 30, 41].”
Fast and Accurate Model Scaling (2021, background): “At intermediate-compute regimes, efficient search [17] or efficient design spaces [22, 23] can still proFigure 1.”
MCUNet: Tiny Deep Learning on IoT Devices (2020, background): “Instead of collecting the Cumulative Distribution Function (CDF) of each satisfying network’s accuracy [37], which is computationally heavy due to tremendous training, we only collect the CDF of FLOPs (see Figure 2).”
Rep ViT: Revisiting Mobile CNN From ViT Perspective (2023, background): “Previous works [41, 42] have shown that the utilization of more blocks in the third stage confers a favorable balance between the accuracy and speed, which is generally adopted by lightweight ViTs.”