FLASH: Fast Bayesian Optimization for Data Analytic Pipelines

By Yuyu Zhang, Mohammad Taha Bahadori, Hang Su, Jimeng Sun. 2016

35 citations recorded by the source corpus.

Modern data science relies on data analytic pipelines to organize interdependent computational steps. Such analytic pipelines often involve different algorithms across multiple steps, each with its own hyperparameters. To achieve the best performance, it is often critical to select optimal algorithms and to set appropriate hyperparameters, which requires large computational efforts. Bayesian optimization provides a principled way for searching optimal hyperparameters for a single algorithm. However, many challenges remain in solving pipeline optimization problems with high-dimensional and highly conditional search space. In this work, we propose Fast LineAr SearcH (FLASH), an efficient method for tuning analytic pipelines. FLASH is a two-layer Bayesian optimization framework, which firstly uses a parametric model to select promising algorithms, then computes a nonparametric model to fine-tune hyperparameters of the promising algorithms. FLASH also includes an effective caching algorithm which can further accelerate the search process. Extensive experiments on a number of benchmark datasets have demonstrated that FLASH significantly outperforms previous state-of-the-art methods in both search speed and accuracy. Using 50% of the time budget, FLASH achieves up to 20% improvement on test error rate compared to the baselines. FLASH also yields state-of-the-art performance on a real-world application for healthcare predictive modeling.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Benchmark and Survey of Automated Machine Learning Frameworks (2019, background): “…problem formulations only consider selecting and tuning a single algorithm (e.g., Escalante et al., 2009; Bergstra et al., 2011) or a linear sequence of algorithms with (arbitrary but) fixed length (e.g., Thornton et al., 2013; Zhang et al., 2016; Alaa & Van Der Schaar, 2018; Hutter et al., 2018a).”
Bayesian Optimization with Tree-structured Dependencies (2017, methodology): “Finally, Zhang et al. (2016) proposed a dedicated approach to tune data analytic pipelines, via a two-layer Bayesian optimization framework.”
Auto-sklearn: Efficient and Robust Automated Machine Learning (2019, methodology): “Since the publication of the original NIPS paper [14] Auto-sklearn has become a standard baseline for new approaches to automated machine learning such as FLASH [39], RECIPE [32], Hyperband [25], Auto-Prognosis [2], ML-PLAN [28], Auto-Stacker [10] and AlphaD3M [11].”
Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning (2020, background): “To improve performance one can prune the configuration space to reduce space to search through (Zhang et al., 2016), split the configuration space into smaller, more managable subspaces (Alaa and van der Schaar, 2018; Liu et al., 2020), or heavily employ expert knowledge (LeDell and Poirier, 2020).”