Multinomial Distribution Learning for Effective Neural Architecture Search

By Xiawu Zheng, Rongrong Ji, Lang Tang, Baochang Zhang, Jianzhuang Liu, Qi Tian. 2019

100 citations recorded by the source corpus.

Architectures obtained by Neural Architecture Search (NAS) have achieved highly competitive performance in various computer vision tasks. However, the prohibitive computation demand of forward-backward propagation in deep neural networks and searching algorithms makes it difﬁcult to apply NAS in practice. In this paper, we propose a Multinomial Distribution Learning for extremely effective NAS, which considers the search space as a joint multinomial distribution, i.e., the operation between two nodes is sampled from this distribution, and the optimal network structure is obtained by the operations with the most likely probability in this distribution. Therefore, NAS can be transformed to a multinomial distribution learning problem, i.e., the distribution is optimized to have a high expectation of the performance. Besides, a hypothesis that the performance ranking is consistent in every training epoch is proposed and demonstrated to further accelerate the learning process. Experiments on CIFAR-10 and ImageNet demonstrate the effectiveness of our method. On CIFAR-10, the structure searched by our method achieves 2.55% test error, while being 6.0× (only 4 GPU hours on GTX1080Ti) faster compared with state-of-the-art NAS algorithms. On ImageNet, our model achieves 74% top1 accuracy under MobileNet settings (MobileNet V1/V2), while being 1.2× faster with measured GPU latency. Test code with pre-trained models are available at https: //github.com/tanglang96/MDENAS

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Comprehensive Survey of Neural Architecture Search (2020, methodology): “MdeNAS [49] has done a lot of experiments to verify this hypothesis, according to which the early performance of candidate architectures can be used to quickly and easily obtain their relative performance rankings, thereby speeding up the search process of the network architecture.”
GreedyNAS: Towards Fast One-Shot NAS With Greedy Supernet (2020, methodology): “To use the gradientbased optimizers for direct searching, an real-valued categorical distribution (architecture parameter) is usually introduced in the supernet, and can be thus jointly learned with the supernet weights, such as DARTS [21], FBNet [38] and MdeNAS [42].”
FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search (2019, neutral): “474 [50].”
Prune and Replace NAS (2019, methodology): “The concept of sharing weights across different architecture configurations has been introduced by ENAS [10] and is now a core component of many recent NAS algorithms [10]–[15].”
One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting (2020, background): “6 presents the Kendall Tau (τ ) metric [25], [70] of the architecture rankings based on weight sharing and retraining.”
Overcoming Multi-Model Forgetting in One-Shot NAS With Diversity Maximization (2020, background): “Figure 3 presents the Kendall Tau ( τ ) metric [16, 36] the normal loss function presents poor supernet predictive ability, which obtains τ = 0 .”
Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search (2019, background): “Some of them extend its use in higher-level architecture search spaces with performance awareness in mind [4, 29], some learn a stochastic distribution instead of architectural parameters [29, 30, 33, 7, 8], and others offer remedies on discovering its lack of robustness [21, 5, 16, 13, 32].”