Auto-Keras: An Efficient Neural Architecture Search System
By Haifeng Jin, Qingquan Song, Xia Hu. 2018
924 citations recorded by the source corpus.
Neural architecture search (NAS) has been proposed to automatically tune deep neural networks, but existing search algorithms, e.g., NASNet, PNAS, usually suffer from expensive computational cost. Network morphism, which keeps the functionality of a neural network while changing its neural architecture, could be helpful for NAS by enabling more efficient training during the search. In this paper, we propose a novel framework enabling Bayesian optimization to guide the network morphism for efficient neural architecture search. The framework develops a neural network kernel and a tree-structured acquisition function optimization algorithm to efficiently explores the search space. Extensive experiments on real-world benchmark datasets have been done to demonstrate the superior performance of the developed framework over the state-of-the-art methods. Moreover, we build an open-source AutoML system based on our method, namely Auto-Keras. The code and documentation are available at https://autokeras.com. The system runs in parallel on CPU and GPU, with an adaptive search strategy for different GPU memory limits.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Frank Hutter
- Yoshua Bengio
- Quoc V. Le
- Sergey Levine
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Dacheng Tao
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Alan L. Yuille
- Jitendra Malik
- Bernt Schiele
- Ming-Hsuan Yang
In Efficiency & Compression
- Marius Thomas Lindauer
- Frank Hutter
- Jiancheng Yang
- Bingbing Ni
- Martin Jaggi
- Corey Nolet
- Joshua Patterson
- R. Shi
In Neural Architecture Search & AutoML
- Xia Hu
- Marius Thomas Lindauer
- Qingquan Song
- Frank Hutter
- Yi-Wei Chen
- Corey Nolet
- Joshua Patterson
- Sebastian Raschka
Citations in the corpus
- AutoML: A Survey of the State-of-the-Art (2019, background): “[20] propose a framework that enables Bayesian optimization to guide the network morphism for an efficient neural architecture search.”
- Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, background): “…of experts experience, automated machine learning (AutoML) has been proposed as a burgeoning technology to design and train neural networks automatically, at the cost of computational resources (Feurer et al., 2015; Katz et al., 2016; Bello et al., 2017; Zoph and Le, 2016; Jin et al., 2018).”
- On Robustness of Neural Architecture Search Under Label Noise (2020, methodology): “…current existing works could be divided into the complete architecture search space (Real et al., 2017; Zoph and Le, 2017; Kandasamy et al., 2018; Jin et al., 2019) and the cell search space (Cai et al., 2018, 2019; Luo et al., 2018; Pham et al., 2018; Zoph et al., 2018; Liu et al., 2019; Real…”
- Evaluating the Search Phase of Neural Architecture Search (2019, methodology): “Bayesian optimization was first introduced to the neural architecture search field by Kandasamy et al. (2018) and Jin et al. (2019).”
- Best Practices for Scientific Research on Neural Architecture Search (2019, methodology): “Although several well-engineered and flexible AutoDL packages were recently proposed (Jin et al., 2019; O’Malley et al., 2019; Erickson et al., 2020; Zimmer et al., 2020), their main goal is ease of use, rather than allowing clean comparisons of several NAS methods.”
- Multi-objective Neural Architecture Search via Predictive Network Performance Optimization (2019, neutral): “(Kandasamy et al., 2018) and (Jin et al., 2019) both define the heuristic distance between neural architectures and use GPs with defined distence for searching.”
- Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, methodology): “Similarly, Auto-keras (Jin et al., 2019) and Auto-PyTorch (Zimmer et al., 2021) use BO to automate the design choices of deep NNs. TPOT (Olson et al., 2016) automates the design choice of an ML pipeline with genetic algorithms, while a probabilistic grammatical evolution is implemented in AutoGOAL…”
- Challenges in Deploying Machine Learning: A Survey of Case Studies (2020, methodology): “There is a large variety of tools that provide general-purpose implementations of AutoML algorithms, such as Auto-keras [137], Auto-sklearn [138] or TPOT [139].”
- Towards Automated Neural Interaction Discovery for Click-Through Rate Prediction (2020, background): “n different tasks, various performance estimation strategies are proposed, including low-fidelity estimation [24, 38], weight sharing [18, 22], learning curve extrapolation [31], and network morphism [13]. 3 HIERARCHICAL SEARCH SPACE DESIGN An ideal search space should contain sufficient distinct ar”
- Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS (2020, methodology): “While two heuristically-designed kernels are provided in [10] and [11], they can neither be easily adapted to different architecture families nor be further optimized based on data.”
- Parameter Prediction for Unseen Deep Architectures (2021, methodology): “Alternative ways to compare graphs include running expensive graph edit distance (GED) algorithms, infeasible for graphs of our size, or designing task-specific graph kernels [78, 112, 77, 106].”