Resource-Efficient Neural Architect

By Yanqi Zhou, Siavash Ebrahimi, Sercan Ö. Arik, Haonan Yu, Hairong Liu, G. Diamos. 2018

64 citations recorded by the source corpus.

Neural Architecture Search (NAS) is a laborious process. Prior work on automated NAS targets mainly on improving accuracy, but lacks consideration of computational resource use. We propose the Resource-Efficient Neural Architect (RENA), an efficient resource-constrained NAS using reinforcement learning with network embedding. RENA uses a policy network to process the network embeddings to generate new configurations. We demonstrate RENA on image recognition and keyword spotting (KWS) problems. RENA can find novel architectures that achieve high performance even with tight resource constraints. For CIFAR10, it achieves 2.95% test error when compute intensity is greater than 100 FLOPs/byte, and 3.87% test error when model size is less than 3M parameters. For Google Speech Commands Dataset, RENA achieves the state-of-the-art accuracy without resource constraints, and it outperforms the optimized architectures with tight resource constraints.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Neural Architecture Search & AutoML

Citations in the corpus

A Survey on Neural Architecture Search (2019, methodology): “Zhou et al. (2018) propose to use λ (gi (α) , ci) = φ max{0,(gi(α)−ci)/ci} , (22)”
Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours (2019, methodology): “Hardware-aware NAS: Earlier NAS methods focused on maximizing accuracy under FLOPs constraints [22,25], but low FLOP count does not necessarily translate to hardware efficiency [10,18].”
Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks (2018, methodology): “These types of approaches are often called multiobjective or platform-aware neural architecture search (Hsu et al., 2018; Zhou et al., 2018; Tan et al., 2018; Dong et al., 2018).”
Graph HyperNetworks for Neural Architecture Search (2018, methodology): “While earlier NAS methods focused on standard image classiﬁcation and language modeling, recent literature has extended NAS to search for architectures that are computationally efﬁcient (Tan et al., 2018; Dong et al., 2018; Hsu et al., 2018; Elsken et al., 2018; Zhou et al., 2018).”
ChamNet: Towards Efficient Network Design Through Platform-Aware Model Adaptation (2018, background): “Most of them optimize NNs based on FLOPs, which is an often-used proxy [37, 35].”
MnasNet: Platform-Aware Neural Architecture Search for Mobile (2018, methodology): “Closely related to our work is MONAS [12], DPP-Net [3], RNAS [34] and Pareto-NASH [4] which attempt to optimize multiple objectives, such as model size and accuracy, while searching for CNNs, but their search process optimizes on small tasks like CIFAR.”
Learnable Embedding Space for Efficient Neural Architecture Compression (2019, background): “Some existing works in NAS are focused on searching for architectures that not only can achieve high performance but also respect some resource or computation constraints (Ashok et al., 2018; Tan et al., 2018; Zhou et al., 2018; Dong et al., 2018; Hsu et al., 2018; Elsken et al., 2018a).”
CARS: Continuous Evolution for Efficient Neural Architecture Search (2019, methodology): “Compared to other multi-objective methods like RENA [56], NSGANet [30] and LEMONADE [10], our searched architectures also show superior performance over these methods.”
RC-DARTS: Resource Constrained Differentiable Architecture Search (2019, background): “[41] modifies the reward function to penalize searched models when constraints are not met.”
Searching for Fast Model Families on Datacenter Accelerators (2021, methodology): “When designing fast models for inference with NAS, previous work employed multi-objective search [56, 18, 13, 29, 64, 27, 12, 21, 39, 16] to consider accuracy together with performance/efficiency.”
Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization (2019, methodology): “Hardware-aware NAS: Earlier NAS methods focused on maximizing accuracy under FLOPs constraints [19], [28], but low FLOP count does not necessarily translate to hardware efficiency [29], [30].”
InstaNAS: Instance-aware Neural Architecture Search (2018, methodology): “[31] use FLOPs and the number of parameters as the proxies of computational costs; Kim et al.”
Machine Learning at Facebook: Understanding Inference at the Edge (2019, methodology): “Looking forward, methods to improve architecture search, including techniques, such as BayesOpt [37, 38], AutoML [39] and [40], are of important interest.”
The need for uncertainty quantification in machine-assisted medical decision making (2019, background): “…for the analysis of images, recurrent neural networks (RNNs) for analysing time series and prediction, and sequence recognizers (for example, long short-term memory units (LSTMs) 15 ) for the analysis of text, though the architecture of the network is itself often the subject of exploration 16 .”