Practical Block-Wise Neural Network Architecture Generation
By Zhaobai Zhong, Junjie Yan, Wei Wu, Jing Shao, Chenglin Liu. 2017
531 citations recorded by the source corpus.
Convolutional neural networks have gained a remarkable success in computer vision. However, most usable network architectures are hand-crafted and usually require expertise and elaborate design. In this paper, we provide a block-wise network generation pipeline called BlockQNN which automatically builds high-performance networks using the Q-Learning paradigm with epsilon-greedy exploration strategy. The optimal network block is constructed by the learning agent which is trained sequentially to choose component layers. We stack the block to construct the whole auto-generated network. To accelerate the generation process, we also propose a distributed asynchronous framework and an early stop strategy. The block-wise generation brings unique advantages: (1) it performs competitive results in comparison to the hand-crafted state-of-the-art networks on image classification, additionally, the best network generated by BlockQNN achieves 3.54% top-1 error rate on CIFAR-10 which beats all existing auto-generate networks. (2) in the meanwhile, it offers tremendous reduction of the search space in designing networks which only spends 3 days with 32 GPUs, and (3) moreover, it has strong generalizability that the network built on CIFAR also performs well on a larger-scale ImageNet dataset.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neurology, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Yi Yang
- Chunhua Shen
- Trevor Darrell
- Alan L. Yuille
In Neurology
In Neural Architecture Search & AutoML
Citations in the corpus
- A Comprehensive Survey of Neural Architecture Search (2020, methodology): “3 2,000 Block-QNN [32] CVPR18 ✓ 3.”
- AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, methodology): “The stacking is fixed and the block is free to combine standard neural network layers into patterns that optimize the accuracy of the model [102, 100].”
- Real-Time Federated Evolutionary Neural Architecture Search (2020, background): “Much work has been done that uses reinforcement learning (RL) for NAS [8], [9], [31]–[33].”
- AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations (2020, background): “[2, 16, 25, 37] propose searching convolution cells which could be stacked repeatedly.”
- Understanding and Robustifying Differentiable Architecture Search (2019, methodology): “We investigate two ways of regularizing the DARTS search procedure, namely via data augmentation and L 2 regularization in the inner objective.”
- Memory-Efficient Hierarchical Neural Architecture Search for Image Denoising (2019, background): “, policy gradients [47, 36] and Q-learning [44], to train a recurrent neural network that acts as a meta-controller to generate potential architectures—typically encoded as sequences—by exploring a predefined search space.”
- AutoGAN: Neural Architecture Search for Generative Adversarial Networks (2019, methodology): “For the optimization algorithm, popular options include reinforcement learning [4, 71, 69, 72], evolutionary algorithm [60], Bayesian optimization [24], random search [7], and gradient-based optimization methods [38, 1].”
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search (2019, background): “To reduce the computation cost, researchers started to search for blocks or cells (Zhong et al., 2018; Zoph et al., 2018) instead of the entire network, and consequently, managed to reduce the overall computational costs by a factor of 7 .”
- Evolving Robust Neural Architectures to Defend from Adversarial Attacks (2019, methodology): “ be employed to explore it. Some widely used search strategies for AAS are: Random Search, Bayesian Optimization [15], Evolutionary Methods [29],[41],[17],[28], Reinforcement Learning [44], [3], [6], [43],[7], [27] and Gradient Based Methods [4], [19], [18]. Finally, a performance estimation (usuall”
- BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, background): “Reinforcement learning based NAS (Baker et al., 2017; Zoph & Le, 2017; Zhong et al., 2018; Zoph *Equal contribution 1Department of Cognitive Robotics, Delft University of Technology, Netherlands 2Department of Computer Science, University College London, UK.”
- A Survey on Neural Architecture Search (2019, methodology): “Zhong et al. (2018) also use a Q-learning-based algorithm to search for network architectures.”
- Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation (2019, background): “possibility of searching for network building blocks, or socalled cells with reinforcement learning (RL) [36, 38] and evolutionary algorithm (EA) [33, 23].”
- NAS-Unet: Neural Architecture Search for Medical Image Segmentation (2019, background): “However, since NASNet [37] successfully stacks the cell together on ImageNet, more recent works [20], [22], [23], [45], [46] prefer to search the repeatable cell structure, but keep the backbone network fixed at first.”
- You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization (2018, background): “Despite their impressive performance, the search processes are incredibly resource-hungry and unpractical for large datasets like ImageNet, though some acceleration methods have been proposed (Zhong et al. (2018); Pham et al. (2018)).”
- ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware (2018, background): “1 1 INTRODUCTION Neural architecture search (NAS) has demonstrated much success in automating neural network architecture design for various deep learning tasks, such as image recognition (Zoph et al., 2018; Cai et al., 2018a; Liu et al., 2018a; Zhong et al., 2018) and language modeling (Zoph & Le, ”
- Evolutionary-Neural Hybrid Agents for Architecture Search (2018, background): “Some works explicitly address resource efficiency [28, 31, 46, 49], which is crucial, as architecture search is known to require a large amount of resources [50].”
- Path-Level Network Transformation for Efficient Architecture Search (2018, methodology): “This scheme has also been incorporated in Zhong et al. (2017) and Liu et al. (2018).”