Understanding Architectures Learnt by Cell-based Neural Architecture Search

By Yao Shu, Wei Wang, Shaofeng Cai. 2019

93 citations recorded by the source corpus.

Neural architecture search (NAS) generates architectures automatically for given tasks, e.g., image classification and language modeling. Recently, various NAS algorithms have been proposed to improve search efficiency and effectiveness. However, little attention is paid to understand the generated architectures, including whether they share any commonality. In this paper, we analyze the generated architectures and give our explanations of their superior performance. We firstly uncover that the architectures generated by NAS algorithms share a common connection pattern, which contributes to their fast convergence. Consequently, these architectures are selected during architecture search. We further empirically and theoretically show that the fast convergence is the consequence of smooth loss landscape and accurate gradient information conducted by the common connection pattern. Contracting to universal recognition, we finally observe that popular NAS architectures do not always generalize better than the candidate architectures, encouraging us to re-think about the state-of-the-art NAS algorithms.

Subfields: Artificial Intelligence, Biophysics, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Electrical and Electronic Engineering

Citations in the corpus

A Comprehensive Survey of Neural Architecture Search (2020, background): “Therefore, a lot of work has emerged on how to reduce the amount of calculation and accelerate the search of the network architecture [18–20, 48, 49, 52, 84, 105].”
One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting (2020, background): “NEURAL architecture search (NAS) has re7cently attracted massive interest from the deep learning community because experts do not have inordinate amounts of time and labor designing neural networks [13], [18], [29], [35], [42], [47], [50], [71].”
NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size (2020, background): “Some works [21], [22] use topology to indicate the connectivity pattern of architecture.”
Dynamic Routing Networks (2019, methodology): “Following conventions [20, 39, 30], we apply SGD with Nesterov-momentum 0.”
Does Unsupervised Architecture Representation Learning Help Neural Architecture Search? (2020, background): “identity, max-pooling) which are often preferred early on in the search, resulting in lower final accuracies [18, 19, 20, 21].”
NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search (2020, unknown): “This practice impedes assertions about the statistical signiﬁcance of the reported results, recently brought into focus by several authors (Yang et al., 2019; Lindauer & Hutter, 2019; Shu et al., 2020; Yu et al., 2020).”