XNAS: Neural Architecture Search with Expert Advice

By Niv Nayman, Asaf Noy, Tal Ridnik, Itamar Friedman, Rong Jin, Lihi Zelnik-Manor. 2019

135 citations recorded by the source corpus.

This paper introduces a novel optimization method for differential neural architecture search, based on the theory of prediction with expert advice. Its optimization criterion is well fitted for an architecture-selection, i.e., it minimizes the regret incurred by a sub-optimal selection of operations. Unlike previous search relaxations, that require hard pruning of architectures, our method is designed to dynamically wipe out inferior architectures and enhance superior ones. It achieves an optimal worst-case regret bound and suggests the use of multiple learning-rates, based on the amount of information carried by the backward gradients. Experiments show that our algorithm achieves a strong performance over several image classification datasets. Specifically, it obtains an error rate of 1.6% for CIFAR-10, 24% for ImageNet under mobile settings, and achieves state-of-the-art results on three additional datasets.

Subfields: Artificial Intelligence, Management Science and Operations Research, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Neural Architecture Search & AutoML

Citations in the corpus

MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation (2020, methodology): “Moreover, recent proposed NAS methods achieve a higher accuracy than our method [20, 11, 2].”
A Comprehensive Survey of Neural Architecture Search (2020, methodology): “e DARTS more robust. Finally, [110] proposed several simple variants and achieved good generalization performance. Although we have done many reviews, there are still many improvements based on DARTS [113, 151]. In this section, we provide a comprehensive and systematic overview of optimized NAS wor”
NAS evaluation is frustratingly hard (2019, methodology): “25 p.p. below state-of-the-art (Nayman et al., 2019).”
DARTS+: Improved Differentiable Architecture Search with Early Stopping (2019, neutral): “2 gradient XNAS [24] 24.”
Understanding Architectures Learnt by Cell-based Neural Architecture Search (2019, background): “…algorithms are proposed to improve the effectiveness (Zoph et al., 2018; Real et al., 2019; Tan et al., 2019; Cai et al., 2019) and the efﬁciency (Pham et al., 2018; Liu et al., 2019; Xie et al., 2019b; Luo et al., 2018; Nayman et al., 2019; Akimoto et al., 2019) of the architecture search.”
Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search (2019, background): “Some of them extend its use in higher-level architecture search spaces with performance awareness in mind [4,36], some learn a stochastic distribution instead of architectural parameters [36,37,40,11,12], and others offer remedies on discovering its lack of robustness [26,6,21,18,39].”
Overcoming Multi-Model Forgetting in One-Shot NAS With Diversity Maximization (2020, methodology): “Several state-of-the-art One-Shot methods utilize continuous relaxation to transform the discrete architecture into continuous space A θ with parameters θ to further improve the efﬁciency [23, 11, 33, 26].”
StacNAS: Towards Stable and Consistent Differentiable Neural Architecture Search (2019, unknown): “(2018); Real et al. (2018); Liu et al.”
Improving One-Shot NAS by Suppressing the Posterior Fading (2019, methodology): “Other relevant works are ASAP [26] and XNAS [25], which introduce pruning during the training of over-parameterized networks to improve the efficiency of NAS.”
Differentiable Feature Aggregation Search for Knowledge Distillation (2020, methodology): “work covering all candidate sub-networks named supernet, and then selects the network architecture from the trained supernet. Among the one-shot methods, the dierentiable architecture search (DARTS) [5,22,24,26,36,40] further relaxes the discrete search space to be continuous and couples the archite”
Sample-Efficient Neural Architecture Search by Learning Action Space (2019, methodology): “XNAS [38] and many other existing NAS methods in Table.”
NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search (2020, background): “Many works since then have tried to improve over this paradigm (Liu et al., 2019; Pham et al., 2018; Xu et al., 2020; Dong & Yang, 2019; Zela et al., 2020a; Nayman et al., 2019).”