EAT-NAS: elastic architecture transfer for accelerating large-scale neural architecture search

By Jiemin Fang, Yukang Chen, Xinbang Zhang, Qian Zhang, Chang Huang, Gaofeng Meng, Xinggang Wang. 2019

25 citations recorded by the source corpus.

Neural architecture search (NAS) methods have been proposed to relieve human experts from tedious architecture engineering. However, most current methods are constrained in small-scale search owing to the issue of huge computational resource consumption. Meanwhile, the direct application of architectures searched on small datasets to large datasets often bears no performance guarantee due to the discrepancy between different datasets. This limitation impedes the wide use of NAS on large-scale tasks. To overcome this obstacle, we propose an elastic architecture transfer mechanism for accelerating large-scale NAS (EAT-NAS). In our implementations, the architectures are first searched on a small dataset, e.g., CIFAR-10. The best one is chosen as the basic architecture. The search process on a large dataset, e.g., ImageNet, is initialized with the basic architecture as the seed. The large-scale search process is accelerated with the help of the basic architecture. We propose not only a NAS method but also a mechanism for architecture-level transfer learning. In our experiments, we obtain two final models EATNet-A and EATNet-B, which achieve competitive accuracies of 75.5% and 75.6%, respectively, on ImageNet. Both the models also surpass the models searched from scratch on ImageNet under the same settings. For the computational cost, EAT-NAS takes only fewer than 5 days using 8 TITAN X GPUs, which is significantly less than the computational consumption of the state-of-the-art large-scale NAS methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xinggang Wang
2. Wenyu Liu
3. Qian Zhang
4. Yuan Li
5. Yuzhu Sun
6. Jiemin Fang
7. Kangjian Peng

In Computer Vision and Pattern Recognition

1. Wenyu Liu
2. Xinggang Wang
3. Qian Zhang
4. Jiemin Fang
5. Yuan Li
6. Yuzhu Sun
7. Kangjian Peng
8. Luc Van Gool

Citations in the corpus

• Fast Neural Network Adaptation via Parameter Remapping and Architecture Search (2020, methodology): “Moreover, some NAS works (Pham et al., 2018; Fang et al., 2019a; Elsken et al., 2019) apply parameters sharing on child models to accelerate the search process.”
• Densely Connected Search Space for More Flexible Neural Architecture Search (2019, background): “Some works [4, 14, 45, 9] simplify the search space by searching for the expansion ratios and kernel sizes of MBConv layers.”