Path-Level Network Transformation for Efficient Architecture Search

By Han Cai, Jiacheng Yang, Weinan Zhang, Song Han, Yong Yu. 2018

221 citations recorded by the source corpus.

We introduce a new function-preserving transformation for efficient neural architecture search. This network transformation allows reusing previously trained networks and existing successful architectures that improves sample efficiency. We aim to address the limitation of current network transformation operations that can only perform layer-level architecture modifications, such as adding (pruning) filters or inserting (removing) a layer, which fails to change the topology of connection paths. Our proposed path-level transformation operations enable the meta-controller to modify the path topology of the given network while keeping the merits of reusing weights, and thus allow efficiently designing effective structures with complex path topologies like Inception models. We further propose a bidirectional tree-structured reinforcement learning meta-controller to explore a simple yet highly expressive tree-structured architecture space that can be viewed as a generalization of multi-branch architectures. We experimented on the image classification datasets with limited computational resources (about 200 GPU-hours), where we observed improved parameter efficiency and better test results (97.70% test accuracy on CIFAR-10 with 14.3M parameters and 74.6% top-1 accuracy on ImageNet in the mobile setting), demonstrating the effectiveness and transferability of our designed architectures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “This is a similar growing criterion adopted in previous works [3, 4, 9].”
Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, methodology): “However, also for larger models, LEMONADE is competitive to methods that require signiﬁcantly more computational resources (Zoph et al., 2018) or start their search with non-trivial architectures (Cai et al., 2018b; Dong et al., 2018).”
A Comprehensive Survey of Neural Architecture Search (2020, background): “For example: EAS [50], Path-level EAS [56] and N2N learning [51], etc.”
AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations (2020, methodology): “For DLRS, (i) embedding layer: we select N = 3 sizes of embedding dimension [ 2 , 16 , 128 ] , thus dimension of transformed embeddings is 128.”
On Robustness of Neural Architecture Search Under Label Noise (2020, background): “, 2019), reinforcement learning (Zoph and Le, 2017; Cai et al., 2018; Pham et al., 2018; Zoph et al., 2018), evolutionary algorithms (Real et al.”
Once for All: Train One Network and Specialize it for Efficient Deployment (2019, background): “Early NAS methods (Zoph et al., 2018; Real et al., 2019; Cai et al., 2018b) search for high-accuracy architectures without taking hardware efﬁciency into consideration.”
Evolving Robust Neural Architectures to Defend from Adversarial Attacks (2019, methodology): “Moreover, many architecture searches focus primarily on the hyperparameter search while using architecture search spaces around previously hand-crafted architecture such as DenseNet [7] which are proved to be vulnerable to adversarial attacks.”
BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, methodology): “For the detailed structure of the tree-cell, we refer to (Cai et al., 2018).”
A Survey on Neural Architecture Search (2019, methodology): “Zhou et al. (2018) propose to use λ (gi (α) , ci) = φ max{0,(gi(α)−ci)/ci} , (22) where φ is a penalization constant in the range of 0 to 1 and use a similar approach to Cai et al. (2018a) to find the final architecture.”
Random Search and Reproducibility for Neural Architecture Search (2019, background): “ucture of neural networks to provide even cheaper, heuristic estimates of quality. Many of these methods center around sharing and reuse: network morphisms build upon previously trained architectures [6, 11, 23]; hypernetworks and performance prediction encode information from previously seen archit”
ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware (2018, methodology): “…2.31 ENAS + c/o (Pham et al., 2018) 4.6M 2.89 DARTS + c/o (Liu et al., 2018c) 3.4M 2.83 NASNet-A + c/o (Zoph et al., 2018) 27.6M 2.40 PathLevel EAS + c/o (Cai et al., 2018b) 14.3M 2.30 AmoebaNet-B + c/o (Real et al., 2018) 34.9M 2.13 Proxyless-R + c/o (ours) 5.8M 2.30 Proxyless-G + c/o (ours) 5.7M”
Neural Input Search for Large Scale Recommendation Models (2019, background): “[16, 20, 11, 4] propose searching convolution cells, which are later stacked repeatedly into a deep network.”
HAQ: Hardware-Aware Automated Quantization With Mixed Precision (2018, background): “[1] introduced the path-level network transformation to effectively search the tree-structured architecture space.”
Probabilistic Neural Architecture Search (2019, background): “A more recent line of work has focused on sharing weights across multiple architectures (Pham et al., 2018; Cai et al., 2018a).”
Taking Human out of Learning Applications: A Survey on Automated Machine Learning (2018, background): “, [7], [20], [40], [54], [55], [56], [57], [58], [59], [60], [61], [62], [72]).”
Deep neural network architecture search using network morphism (2019, result): “The way we created concatenation and add nodes is similar as in [21].”