DSNAS: Direct Neural Architecture Search Without Parameter Retraining

By Shou-Yong Hu, Sirui Xie, Hehui Zheng, Chunxiao Liu, Jianping Shi, Xunying Liu, Dahua Lin. 2020

146 citations recorded by the source corpus.

If NAS methods are solutions, what is the problem? Most existing NAS methods require two-stage parameter optimization. However, performance of the same architecture in the two stages correlates poorly. In this work, we propose a new problem definition for NAS, task-specific end-to-end, based on this observation. We argue that given a computer vision task for which a NAS method is expected, this definition can reduce the vaguely-defined NAS evaluation to i) accuracy of this task and ii) the total computation consumed to finally obtain a model with satisfying accuracy. Seeing that most existing methods do not solve this problem directly, we propose DSNAS, an efficient differentiable NAS framework that simultaneously optimizes architecture and parameters with a low-biased Monte Carlo estimate. Child networks derived from DSNAS can be deployed directly without parameter retraining. Comparing with two-stage methods, DSNAS successfully discovers networks with comparable accuracy (74.4\%) on ImageNet in 420 GPU hours, reducing the total time by more than 34\%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Quoc V. Le
2. Jiahui Yu
3. Thomas S. Huang
4. Hanxiao Liu
5. Mingxing Tan
6. Gabriel Bender
7. Pieter-Jan Kindermans
8. Xiaodan Song

In Computer Vision and Pattern Recognition

1. Quoc V. Le
2. Thomas S. Huang
3. Luc Van Gool
4. Dacheng Tao
5. Jiahui Yu
6. Ming-Hsuan Yang
7. Trevor Darrell
8. Xiaogang Wang

In Neural Architecture Search & AutoML

1. Hanxiao Liu
2. Quoc V. Le
3. Pieter-Jan Kindermans
4. Yonggan Fu
5. Mingxing Tan
6. Jiahui Yu
7. Thomas S. Huang
8. Gabriel Bender

Citations in the corpus

• BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models (2020, methodology): “Each architecture in the search space uses only a subset of the operations in the one-shot model; these child models can be efficiently ranked by using the shared weights to estimate their relative accuracies [2, 4, 6, 19,23,26,34–36].”
• Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search (2020, background): “In contrast, the one-level optimization has unbiased gradient estimation [26] and its potential over-fitting problem can be alleviated by the randomness introduced by the Gumbel Softmax function [36] as widely adopted in [82, 34, 6].”