Automated Machine Learning: From Principles to Practices

By Quanming Yao, Mengshuo Wang, Hugo Jair Escalante, Isabelle M Guyon, Yi-Qi Hu, Yu-Feng Li, Wei-Wei Tu, Qiang Yang, Yang Yu. 2018

264 citations recorded by the source corpus.

Machine learning (ML) methods have been developing rapidly, but configuring and selecting proper methods to achieve a desired performance is increasingly difficult and tedious. To address this challenge, automated machine learning (AutoML) has emerged, which aims to generate satisfactory ML configurations for given tasks in a data-driven way. In this paper, we provide a comprehensive survey on this topic. We begin with the formal definition of AutoML and then introduce its principles, including the bi-level learning objective, the learning strategy, and the theoretical interpretation. Then, we summarize the AutoML practices by setting up the taxonomy of existing works based on three main factors: the search space, the search algorithm, and the evaluation strategy. Each category is also explained with the representative methods. Then, we illustrate the principles and practices with exemplary applications from configuring ML pipeline, one-shot neural architecture search, and integration with foundation models. Finally, we highlight the emerging directions of AutoML and conclude the survey.

Subfields: Artificial Intelligence, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tong Zhang
2. Quanming Yao
3. Xia Hu
4. Mu Li
5. Ling Shao
6. Deng-Ping Fan
7. Qingquan Song
8. Alex Smola

In Neural Architecture Search & AutoML

1. Quanming Yao
2. Yi-Wei Chen
3. Zhangyang Wang
4. Xia Hu
5. Tong Zhang
6. Qingquan Song
7. Tong Yu
8. Hong Zhu

Citations in the corpus

• Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “This method is especially suitable for HPO among all AutoML tasks because it is easy to use for quantification (Yao et al., 2018).”
• AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, neutral): “Reviews provide more thorough background [21, 81, 95].”
• AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data (2020, background): “Numerous AutoML frameworks have recently emerged to solve this problem, as evidenced by a flurry of survey articles (Yao et al., 2018; He et al., 2019; Truong et al., 2019; Guyon et al., 2019; Gijsbers et al., 2019; Zöller & Huber, 2019).”
• Concealed Object Detection (2021, methodology): “Therefore, neural architecture search techniques, such as the popular automated machine learning [106], offer a potential direction.”
• A Survey of Graph Neural Networks for Recommender Systems: Challenges, Methods, and Directions (2021, unknown): “Recently, AutoML (Automated Machine Learning) [221] has been proposed, which can automatically design appropriate models for specific tasks.”
• Propagation Model Search for Graph Neural Networks (2020, background): “Neural architecture search (NAS) (Elsken, Metzen, and Hutter 2019; Yao et al. 2018) has become a promising way to automatically discover novel neural network architectures.”
• Techniques for Automated Machine Learning (2019, methodology): “Several surveys on AutoMHT [101], AutoDL [24; 94; 37], and general AutoML [72] have been published from different perspectives, unveiling the mysterious art behind AutoML.”
• Differentiable Neural Architecture Search via Proximal Iterations (2019, background): “Recently, the neural architecture search (NAS) [47, 3] has been developed as a promising approach to replace human experts on designing architectures, which can find networks with fewer parameters and better performance than hand-crafted ones [42, 17].”
• AOGNets: Compositional Grammatical Architectures for Deep Learning (2017, background): “Related work on neural architecture search is referred to the survey papers [8, 64].”
• AOGNets: Deep AND-OR Grammar Networks for Visual Recognition (2017, background): “-scale datasets such as the ImageNet benchmark [38]. We focus on hand-crafted architectures in this section. Related work on automatic neural architecture search is referred to the nice survey papers [4,53]. Hand-crafted network architectures. After more than 20 years since the seminal work 5-layer ”
• Learning to Optimize: A Primer and A Benchmark (2021, background): “The term “AutoML” broadly refers to the automation of any step(s) in the ML lifecycle.”
• Federated learning for predicting clinical outcomes in patients with COVID-19 (2021, methodology): “Future approaches may incorporate automated hyperparam-eter searching 55 , neural architecture search 56 and other automated machine learning 57 approaches to find the optimal training parameters for each client site more efficiently.”