Understanding Neural Architecture Search Techniques

By George Adam, Jonathan Lorraine. 2019

46 citations recorded by the source corpus.

Automatic methods for generating state-of-the-art neural network architectures without human experts have generated significant attention recently. This is because of the potential to remove human experts from the design loop which can reduce costs and decrease time to model deployment. Neural architecture search (NAS) techniques have improved significantly in their computational efficiency since the original NAS was proposed. This reduction in computation is enabled via weight sharing such as in Efficient Neural Architecture Search (ENAS). However, recently a body of work confirms our discovery that ENAS does not do significantly better than random search with weight sharing, contradicting the initial claims of the authors. We provide an explanation for this phenomenon by investigating the interpretability of the ENAS controller's hidden state. We find models sampled from identical controller hidden states have no correlation with various graph similarity metrics, so no notion of structural similarity is learned. This failure mode implies the RNN controller does not condition on past architecture choices. Lastly, we propose a solution to this failure mode by forcing the controller's hidden state to encode pasts decisions by training it with a memory buffer of previously sampled architectures. Doing this improves hidden state interpretability by increasing the correlation between controller hidden states and graph similarity metrics.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Wanli Ouyang
2. Miao Zhang
3. Xiaojun Chang
4. Xiangxiang Chu
5. Bo Zhang
6. Junjie Yan
7. Shirui Pan
8. Huiqi Li

Citations in the corpus

• FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search (2019, background): “They behaved poorly in previous works like [15, 2] just because they are unfairly trained, hence it was due to a rich-get-richer phenomenon, as noted in [1].”
• Convolutional neural network: a review of models, methodologies and applications to object detection (2019, methodology): “In progressive neural architecture search (PNAS) [109], sequential model-based optimization (SMBO) Table 13 Publications relevant to ResNet”
• Overcoming Multi-Model Forgetting in One-Shot NAS With Diversity Maximization (2020, background): “Adam et al. [1] showed that the RNN controller in One-Shot NAS does not depend on past sampled architectures, which makes its performance the same as random search.”
• Improving One-Shot NAS by Suppressing the Posterior Fading (2019, background): “However, as identified in [1, 9, 3], the problem of one-shot method is that the validation accuracy of the model with shared weights is not predictive to its true performance.”
• One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting (2020, background): “[1] show that the RNN controller in ENAS”