AutoML: A Survey of the State-of-the-Art
By Xin He, Kaiyong Zhao, Xiaowen Chu. Knowledge-Based Systems, 2019
1,830 citations recorded by the source corpus.
Deep learning (DL) techniques have penetrated all aspects of our lives and brought us great convenience. However, building a high-quality DL system for a specific task highly relies on human expertise, hindering the applications of DL to more areas. Automated machine learning (AutoML) becomes a promising solution to build a DL system without human assistance, and a growing number of researchers focus on AutoML. In this paper, we provide a comprehensive and up-to-date review of the state-of-the-art (SOTA) in AutoML. First, we introduce AutoML methods according to the pipeline, covering data preparation, feature engineering, hyperparameter optimization, and neural architecture search (NAS). We focus more on NAS, as it is currently very hot sub-topic of AutoML. We summarize the performance of the representative NAS algorithms on the CIFAR-10 and ImageNet datasets and further discuss several worthy studying directions of NAS methods: one/two-stage NAS, one-shot NAS, and joint hyperparameter and architecture optimization. Finally, we discuss some open problems of the existing AutoML methods for future research.
Subfields: Artificial Intelligence, Neural Architecture Search & AutoML
View Paper: https://arxiv.org/abs/1908.00709
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Neural Architecture Search & AutoML
Citations in the corpus
- The Efficiency Misnomer (2021, methodology): “Aside from being used for comparing models, many architecture search studies add a cost indicator to the loss function as a resource constraint (He et al., 2021b) and account for efficiency.”
- VEGA: Towards an End-to-End Configurable AutoML Pipeline (2020, background): “Automated Machine Learning (AutoML) (He et al., 2019) is an integration of machine learning algorithms that enables developers and users with limited machine learning expertise to train and deploy high-quality models for their own tasks and needs.”
- Evolving Robust Neural Architectures to Defend from Adversarial Attacks (2019, background): “…many architecture searches focus primarily on the hyper-parameter search while using architecture search spaces around previously hand-crafted architecture [Lee et al., 2018; He et al., 2019] such as DenseNet which are proved to be vulnerable to adversarial attacks [Vargas and Kotyan, 2019].”
- Benchmark and Survey of Automated Machine Learning Frameworks (2019, methodology): “Elshawi et al. (2019) and He et al. (2019) focus on automatic neural network architecture search—which is not covered in this survey—and only briefly introduce methods for classic machine learning.”
- Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence (2020, methodology): “Machine learning pipelines often begin with a data preparation step, which typically includes data cleaning, mapping individual fields to data types in preparation for feature engineering, and imputing missing values [54,55].”
- AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data (2020, background): “Numerous AutoML frameworks have recently emerged to solve this problem, as evidenced by a flurry of survey articles (Yao et al., 2018; He et al., 2019; Truong et al., 2019; Guyon et al., 2019; Gijsbers et al., 2019; Zöller & Huber, 2019).”
- Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model (2024, methodology): “Much effort has been put into using machine learning techniques (Bengio et al., 2021; Chen et al., 2022; He et al., 2021; Li et al., 2023a) in automatic algorithm design.”
- Explainable AI (XAI): Core Ideas, Techniques, and Solutions (2022, background): “AutoML has made explainability more important, as it enables the entire data pipeline into a black box (as opposed to just the ML model) [27].”
- TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data (2022, background): “The problem of anomaly detection is becoming increasingly challenging in large-scale databases due to the increasing data modality [18, 28, 54].”
- Evolutionary Architecture Search for Graph Neural Networks (2020, background): “Bayesian Optimization (BO) is a family of algorithms that build a probability model of the objective function determining the best expected neural network architecture [31].”
- Physics-informed neural networks for inverse problems in nano-optics and metamaterials. (2019, background): “[40] Xin He, Kaiyong Zhao, and Xiaowen Chu, “Automl: A survey of the state-of-the-art,” arXiv preprint arXiv:1908.”
- Amazon SageMaker Autopilot: a white box AutoML solution at scale (2020, background): “For more details on the state of the field on automatic machine learning we refer reader to comprehensive surveys [25] and [12].”
- A Probabilistic Framework for Modular Continual Learning (2023, background): “The training data ( X tr , Y tr ) and validation data ( X val , Y val ) for the new task. π ← () , f ← () ; // evaluated paths and their validation performance // Perform Bayesian Optimisation for NT paths with ℓ min pre-trained modules.”