AtomNAS: Fine-Grained End-to-End Neural Architecture Search

By Jieru Mei, Yingwei Li, Xiaochen Lian, Xiaojie Jin, Linjie Yang, Alan L. Yuille, Jianchao Yang. 2019

111 citations recorded by the source corpus.

Designing of search space is a critical problem for neural architecture search (NAS) algorithms. We propose a fine-grained search space comprised of atomic blocks, a minimal search unit much smaller than the ones used in recent NAS algorithms. This search space facilitates direct selection of channel numbers and kernel sizes in convolutions. In addition, we propose a resource-aware architecture search algorithm which dynamically selects atomic blocks during training. The algorithm is further accelerated by a dynamic network shrinkage technique. Instead of a search-and-retrain two-stage paradigm, our method can simultaneously search and train the target architecture in an end-to-end manner. Our method achieves state-of-the-art performance under several FLOPS configurations on ImageNet with a negligible searching cost. We open our entire codebase at: https://github.com/meijieru/AtomNAS.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Alan L. Yuille
2. Quoc V. Le
3. Qi Tian
4. Lingxi Xie
5. Xiaojun Chang
6. Xiangxiang Chu
7. Chen Liang
8. Bo Zhang

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Alan L. Yuille
3. Qi Tian
4. Dacheng Tao
5. Yi Yang
6. Xiaogang Wang
7. Ming-Hsuan Yang
8. Jiashi Feng

In Neural Architecture Search & AutoML

1. Quoc V. Le
2. Bo Zhang
3. Xiaojun Chang
4. Xiangxiang Chu
5. Qi Tian
6. Xin Chen
7. Lingxi Xie
8. Chen Liang

Citations in the corpus

• A Comprehensive Survey of Neural Architecture Search (2020, background): “Therefore, a lot of work has emerged on how to reduce the amount of calculation and accelerate the search of the network architecture [18–20, 48, 49, 52, 84, 105].”
• AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, background): “Mei et al. (2020), on the other hand, perform a fine-grained search over the convolutions of a neural network.”
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search (2019, background): “, 2019), which led to a few improved approaches beyond DARTS (Cai et al., 2019; Chen et al., 2019; Mei et al., 2020).”
• F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization (2022, background): “…and extensively studied in the last decades, resulting in significant improvement on the trade-off between efficiency and accuracy (Han et al., 2015; Liu et al., 2018; Mei et al., 2019; Tanaka et al., 2020; Ma et al., 2020; Mishra et al., 2020; Liang et al., 2021; Jin et al., 2021; Liu et al., 2021).”
• MTL-NAS: Task-Agnostic Neural Architecture Search Towards General-Purpose Multi-Task Learning (2020, background): “Recently, many neural architecture search (NAS) methods have emerged to jointly learn the weights and the network architectures [4, 6, 12, 14, 22, 31, 34, 41, 44, 45, 71].”
• DARTS-: Robustly Stepping out of Performance Collapse Without Indicators (2020, background): “Since then, many subsequent works have been dedicated to accelerating the process (Dong & Yang, 2019b), reducing memory cost (Xu et al., 2020), or fostering its ability such as hardware-awareness (Cai et al., 2019; Wu et al., 2019), finer granularity (Mei et al., 2020) and so on.”
• Zero-Cost Proxies for Lightweight NAS (2021, methodology): “For example, Mei et al. (2020) use channel pruning in their AtomNAS work to arrive at customized multi-kernel-size convolutions (mixconvs as introduced by Tan & Le (2019b)).”
• One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting (2020, background): “NEURAL architecture search (NAS) has re7cently attracted massive interest from the deep learning community because experts do not have inordinate amounts of time and labor designing neural networks [13], [18], [29], [35], [42], [47], [50], [71].”
• Neural Architecture Search for Lightweight Non-Local Networks (2020, background): “[26] proposes an ensemble perspective of the basic block and simultaneously searches and trains the target architecture in the fine-grained search space.”