ParamILS: An Automatic Algorithm Configuration Framework
By Frank Hutter, Holger H. Hoos, Kevin Leyton-Brown, Thomas Stützle. 2014
1,129 citations recorded by the source corpus.
The identification of performance-optimizing parameter settings is an important part of the development and application of algorithms. We describe an automatic framework for this algorithm configuration problem. More formally, we provide methods for optimizing a target algorithm's performance on a given class of problem instances by varying a set of ordinal and/or categorical parameters. We review a family of local-search-based algorithm configuration procedures and present novel techniques for accelerating them by adaptively limiting the time spent for evaluating individual configurations. We describe the results of a comprehensive experimental evaluation of our methods, based on the configuration of prominent complete and incomplete algorithms for SAT. We also present what is, to our knowledge, the first published work on automatically configuring the CPLEX mixed integer programming solver. All the algorithms we considered had default parameter settings that were manually identified with considerable effort. Nevertheless, using our automated algorithm configuration procedures, we achieved substantial and consistent performance improvements.
Subfields: Computational Theory and Mathematics, Computer Networks and Communications, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Theory and Mathematics
- Frank Hutter
- Ponnuthurai Nagaratnam Suganthan
- Holger H. Hoos
- Marco F. Huber
- Marc-André Zöller
- Yaochu Jin
- Xin Yao
- Marius Thomas Lindauer
In Computer Networks and Communications
- Frank Hutter
- Holger H. Hoos
- Kevin Leyton-Brown
- Yuri Malitsky
- Katharina Eggensperger
- Lars Kotthoff
- Marius Thomas Lindauer
- Chris J. Thornton
In Neural Architecture Search & AutoML
- Frank Hutter
- Holger H. Hoos
- Kevin Leyton-Brown
- Matthias Feurer
- Gang Luo
- Marc-André Zöller
- Marco F. Huber
- Marius Thomas Lindauer
Citations in the corpus
- Benchmark and Survey of Automated Machine Learning Frameworks (2019, background): “Following Hutter et al. (2009), Thornton et al. (2013), Swearingen et al. (2017) the hyperparameters ~λ ∈ ΛA(i) can be aggregated in two groups: mandatory hyperparameters always have to be present while conditional hyperparameters depend on the selected value of another hyperparameter.”
- Survey on Automated Machine Learning (2019, background): “Following Hutter et al. (2009); Thornton et al. (2013); Swearingen et al. (2017) the hyperparameters λ ∈ Λ(i) can be aggregated in two groups.”
- Efficient Benchmarking of Hyperparameter Optimizers via Surrogates (2015, background): “All rights reserved. programming (Hutter et al. 2009).”
- Learning to Run Heuristics in Tree Search (2017, methodology): “In the context of MIP, ML has been used to find good parameter configurations for a solver [Hutter et al., 2009], design improved node [Sabharwal et al.”
- Learning to Branch in Mixed Integer Programming (2016, methodology): “In the context of MIP, some recent works propose ML techniques for constructing a portfolio of good parameter configura-tions for a MIP solver, and selecting the best configuration for a given instance (Hutter et al. 2009; Kadioglu et al. ; Xu et al. 2011).”
- Easy Hyperparameter Search Using Optunity (2014, methodology): “Finally, ParamILS is a command-lineonly tuning framework providing iterated local search (Hutter et al., 2009).”
- Hydra-MIP : Automated Algorithm Configuration and Selection for Mixed Integer Programming (2011, methodology): “Because a single run of a randomized configuration procedure might not yield a high-performing parameter configuration, it is common practice to perform multiple runs in parallel and to use the configuration that performs best on the training set [12, 14, 28, 13].”
- Sequential Model-Based Optimization for General Algorithm Configuration (2011, methodology): “The most prominent configuration methods are the racing algorithm F-RACE [5] and our own iterated local search algorithm PARAMILS [7, 8].”
- Solving Mixed Integer Programs Using Neural Networks (2020, background): “Algorithm Configuration (Ansótegui et al. 2009, Hutter et al. 2009, 2011, Ansótegui et al. 2015) aims to find parameter values that improve aggregate performance on a dataset of instances.”
- Efficient Parameter Importance Analysis via Ablation with Surrogates (2017, methodology): “Such models are called empirical performance models (EPMs; Leyton-Brown, Nudelman, and Shoham; Hutter et al. 2009; 2014b).”
- Monte Carlo Tableau Proof Search (2016, methodology): “On the parameter training set, we ran ParamILS (Hutter et al. 2009) with monteCoP, using a problem timeout of 2s.”
- Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms (2012, methodology): “We use k-fold cross-validation [19], which splits the training data into k equal-sized partitions D(1)valid, . . . ,D (k) valid, and sets D(i)train = D \ D (i) valid for i = 1, . . . , k. 1”
- A review of automatic selection methods for machine learning algorithms and hyper-parameter values (2016, methodology): “In addition to making machine learning accessible to layman users, automatic selection methods for hyper-parameter values also facilitate comparing different algorithms based on generalization accuracy through automatically tuning hyper-parameters (Hutter et al. 2009).”
- Efficient Multi-Start Strategies for Local Search Algorithms (2009, methodology): “This is the same type of dominance as used by Hutter et al. (2009) in a slightly different context.”
- Parameter tuning for configuring and analyzing evolutionary algorithms (2011, methodology): “It is an extension to the ParamILS framework [56] that describes a work-flow very similar to a (1 + 1) evolution strategy.”
- Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters (2013, background): “Following [10] and [8], we say that a hyperparameter λ i is conditional on another hyperparameter λ j if λ i is only active if hyperparameter λ j takes values from a given set V i (j) Λ j .”
- Warmstarting of Model-based Algorithm Configuration (2017, methodology): “, its cost), it is known as an empirical performance model (EPM; Leyton-Brown, Nudelman, and Shoham; Hutter et al. 2009; 2014b).”
- Programming by optimization (2012, background): “Furthermore, there are situations in which candidate configurations can be discarded without completing runs that exceed a certain time bound, considering the performance measured for other configurations; such runs can be capped, or terminated when that time bound is reached or exceeded.(19) Note, ”
- Alors: An algorithm recommender system (2017, background): “…in SAT and CSP (Rice, 1976; Leyton-Brown et al., 2003; Epstein et al., 2006; Samulowitz and Memisevic, 2007; Xu et al., 2008),(O´Mahony et al., 2008; Hutter et al., 2009; Pulina and Tacchella, 2010; Kadioglu et al., 2010; Xu et al., 2012a; Malitsky et al., 2013; Malitsky and O´ Sullivan, 2014;…”