Multi-Fidelity Automatic Hyper-Parameter Tuning via Transfer Series Expansion

By Yi-Qi Hu, Yang Yu, Wei-Wei Tu, Qiang Yang, Yuqiang Chen, Wenyuan Dai. 2019

37 citations recorded by the source corpus.

Automatic machine learning (AutoML) aims at automatically choosing the best configuration for machine learning tasks. However, a configuration evaluation can be very time consuming particularly on learning tasks with large datasets. This limitation usually restrains derivative-free optimization from releasing its full power for a fine configuration search using many evaluations. To alleviate this limitation, in this paper, we propose a derivative-free optimization framework for AutoML using multi-fidelity evaluations. It uses many lowfidelity evaluations on small data subsets and very few highfidelity evaluations on the full dataset. However, the lowfidelity evaluations can be badly biased, and need to be corrected with only a very low cost. We thus propose the Transfer Series Expansion (TSE) that learns the low-fidelity correction predictor efficiently by linearly combining a set of base predictors. The base predictors can be obtained cheaply from down-scaled and experienced tasks. Experimental results on real-world AutoML problems verify that the proposed framework can accelerate derivative-free configuration search significantly by making use of the multi-fidelity evaluations.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Neural Architecture Search & AutoML

Citations in the corpus

AutoML: A Survey of the State-of-the-Art (2019, neutral): “Similar to ensemble learning, [117] propose the Transfer Series Expansion (TSE) that constructs an ensemble estimator by linearly combining a series of base low fidelity estimators, hence it solves the problem that a single low fidelity estimator can be badly biased.”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “To solve the problem posed by a small data subset, a transfer series expansion method was proposed on hyper-parameter tuning tasks to accelerate the evaluation with a combination of a set of predictors (Hu et al., 2019).”
Automated Machine Learning: State-of-The-Art and Open Challenges (2019, methodology): “Multi-fidelity optimization is an optimization technique which focuses on decreasing the evaluation cost by combining a large number of cheap low-fidelity evaluations and a small number of expensive high-fidelity evaluation [38, 79, 50].”
Automated Machine Learning: From Principles to Practices (2018, methodology): “As shown in Section 2.3, the training objective in machine learning is usually expressed as a summation of loss over a large set samples.”