Densely Connected Search Space for More Flexible Neural Architecture Search

By Jiemin Fang, Yuzhu Sun, Qian Zhang, Yuan Li, Wenyu Liu, Xinggang Wang. 2019

137 citations recorded by the source corpus.

Neural architecture search (NAS) has dramatically advanced the development of neural network design. We revisit the search space design in most previous NAS methods and find the number and widths of blocks are set manually. However, block counts and block widths determine the network scale (depth and width) and make a great influence on both the accuracy and the model cost (FLOPs/latency). In this paper, we propose to search block counts and block widths by designing a densely connected search space, i.e., DenseNAS. The new search space is represented as a dense super network, which is built upon our designed routing blocks. In the super network, routing blocks are densely connected and we search for the best path between them to derive the final architecture. We further propose a chained cost estimation algorithm to approximate the model cost during the search. Both the accuracy and model cost are optimized in DenseNAS. For experiments on the MobileNetV2-based search space, DenseNAS achieves 75.3% top-1 accuracy on ImageNet with only 361MB FLOPs and 17.9ms latency on a single TITAN-XP. The larger model searched by DenseNAS achieves 76.1% accuracy with only 479M FLOPs. DenseNAS further promotes the ImageNet classification accuracies of ResNet-18, -34 and -50-B by 1.5%, 0.5% and 0.3% with 200M, 600M and 680M FLOPs reduction respectively. The related code is available at https://github.com/JaminFong/DenseNAS.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Comprehensive Survey of Neural Architecture Search (2020, background): “Global search still has its unique research value because it gives the network architecture design a higher degree of freedom [38, 103, 152].”
Fast Neural Network Adaptation via Parameter Remapping and Architecture Search (2020, methodology): “Moreover, some NAS works (Pham et al., 2018; Fang et al., 2019a; Elsken et al., 2019) apply parameters sharing on child models to accelerate the search process.”
One-Shot Neural Architecture Search via Self-Evaluated Template Network (2019, background): “Automatically discovering effective networks has attracted more and more researchers [6, 1, 42, 24, 15, 13].”
A review of convolutional neural network architectures and their optimizations (2022, background): “In the level form (Liu et al. 2019; Fang et al. 2020), the connections of the network as a whole and the way of operation within the connection layer are allowed to search, which can significantly reduce human intervention.”