Automating model search for large scale machine learning

By Evan R. Sparks, Ameet Talwalkar, Daniel Haas, M. Franklin, Michael I. Jordan, Tim Kraska. 2015

166 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Neural Architecture Search & AutoML

Citations in the corpus

Automated Machine Learning: State-of-The-Art and Open Challenges (2019, background): “Tree-Structured Parzen estimator (TPE) has shown great performance for hyper-parameter optimization tasks [13, 12, 35, 37, 115].”
Benchmark and Survey of Automated Machine Learning Frameworks (2019, background): “Various open-source designs for AutoML services have been proposed (e.g., Sparks et al., 2015; Chan, 2017; Swearingen et al., 2017; Koch et al., 2018) but also several commercial solutions exist (e.g., Golovin et al., 2017; Clouder, 2018; H2O.ai, 2018).”
Survey on Automated Machine Learning (2019, methodology): “Various open-source designs for AutoML services have been proposed, e.g., (Sparks et al., 2015; Chan, 2017; Swearingen et al., 2017; Koch et al., 2018), but also several commercial solutions exist, e.g., (Golovin et al., 2017; Clouder, 2018; H2O.ai, 2018).”
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization (2016, methodology): “Hyperband is a general-purpose technique that makes minimal assumptions unlike prior conﬁguration evaluation approaches (Domhan et al., 2015; Swersky et al., 2014; Gy¨orgy and Kocsis, 2011; Agarwal et al., 2011; Sparks et al., 2015; Jamieson and Talwalkar, 2015).”
Tune: A Research Platform for Distributed Model Selection and Training (2018, methodology): “Further, systems such as Spearmint, HyperOpt, and TuPAQ (MLBase) (Sparks et al. (2015)) treat a full trial execution as an atomic unit, which does not allow for intermediate control of trial execution.”
Gandiva: Introspective Cluster Scheduling for Deep Learning (2018, background): “Google Vizier [21], HyperDrive [41], and TuPAQ [45] focus more on the system design of AutoML.”
To Index or Not to Index: Optimizing Exact Maximum Inner Product Search (2017, methodology): “This approach, which we refer to as blocked matrix multiply (BMM) throughout this paper, has been applied to other areas of machine learning (such as model search [32]) and is surprisingly competitive with the state of the art.”
Efficient Hyperparameter Optimization and Infinitely Many Armed Bandits (2016, background): “To overcome these difficulties, Sparks et al. (2015) proposed a halving style algorithm that did not require explicit convergence behavior, and Jamieson & Talwalkar (2015) analyzed a similar algorithm, providing theoretical guarantees as well as encouraging empirical results.”
A review of automatic selection methods for machine learning algorithms and hyper-parameter values (2016, methodology): “partly because the optimization target lacks smoothness (Sparks et al. 2015).”
Towards reliable interactive data cleaning: a user survey and recommendations (2016, methodology): “For instance , active learning is used as part of crowd-sourced label acquisition [17,20] to optimize an operation such as de-duplication; Wrangler automatically generates string extraction rules so that the user only needs to pick from a set of options; and TuPAQ [35] performs automatic hyper-param”
HyperDrive: exploring hyperparameters with POP scheduling (2017, methodology): “Unlike prior work [25] that uses instantaneous accuracy only to identify a fixed set of promising configurations, we incorporate the trajectory of full learning curves and leverage a probabilistic model to”
SystemML: Declarative Machine Learning on Spark (2016, methodology): “Frameworks like MLBase [20, 32] or Columbus [42] further enable declarative ML tasks of model or feature selection, where they optimize both model accuracy and performance of ML algorithms.”
Big data analytics on Apache Spark (2016, background): “A new component calledTuPAQ (Training-supported Predictive Analytic Query Planner) [74,75] automatically finds and trains models for large-scale machine learning.”
Hyperband: Bandit-Based Configuration Evaluation for Hyperparameter Optimization (2016, methodology): “2 Lastly, in an effort avoid heuristics and strong assumptions, Sparks et al. (2015) proposed a halving style algorithm that did not require explicit convergence behavior, and Jamieson & Talwalkar (2015) analyzed a similar algorithm, providing theoretical guarantees and encouraging empirical…”
Democratizing Data Science through Interactive Curation of ML Pipelines (2019, background): “Spark TuPAQ [36] and Hyperband [22] use variations of multi-armed bandit (MAB) algorithm to better allocate computational resources for hyper-parameter tuning.”
Optimus: an efficient dynamic resource scheduler for deep learning clusters (2018, background): “time estimation [56] and data size estimation of SQL queries [53].”
PipeDream: generalized pipeline parallelism for DNN training (2019, methodology): “Guided by prior work, we adjust the learning rate during training to converge to the desired result faster [35, 49] and utilize learning rate warm-up for large global batch sizes [24].”