Efficient Parameter Importance Analysis via Ablation with Surrogates

By André Biedenkapp, Marius Thomas Lindauer, Katharina Eggensperger, Frank Hutter, Chris Fawcett, Holger H. Hoos. 2017

59 citations recorded by the source corpus.

To achieve peak performance, it is often necessary to adjust the parameters of a given algorithm to the class of problem instances to be solved; this is known to be the case for popular solvers for a broad range of AI problems, including AI planning, propositional satisfiability (SAT) and answer set programming (ASP). To avoid tedious and often highly sub-optimal manual tuning of such parameters by means of ad-hoc methods, general-purpose algorithm configuration procedures can be used to automatically find performance-optimizing parameter settings. While impressive performance gains are often achieved in this manner, additional, potentially costly parameter importance analysis is required to gain insights into what parameter changes are most responsible for those improvements. Here, we show how the running time cost of ablation analysis, a well-known general-purpose approach for assessing parameter importance, can be reduced substantially by using regression models of algorithm performance constructed from data collected during the configuration process. In our experiments, we demonstrate speed-up factors between 33 and 14 727 for ablation analysis on various configuration scenarios from AI planning, SAT, ASP and mixed integer programming (MIP).

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, neutral): “The data gathered by BOHB on different budgets could also be used to quantify the importance of hyperparameters (Hutter et al., 2014; Biedenkapp et al., 2017; Golovin et al., 2017; van Rijn & Hutter, 2018).”
Hyperparameter Importance Across Datasets (2017, background): “rinciple. It predicts the performance of a classifier based on a subset of hyperparameters that is initialized empty and greedily filled with the next most important hyperparameter. Ablation Analysis [3, 11] requires a default setting and an optimized setting and calculates a so-called ablation trac”
Warmstarting of Model-based Algorithm Configuration (2017, background): “Another direction for future work is to combine warmstarting with parameter importance analysis (Hutter, Hoos, and Leyton-Brown 2014; Biedenkapp et al. 2017), e.”
Tunability: Importance of Hyperparameters of Machine Learning Algorithms (2018, methodology): “, 2010), which try to minimize the expected generalization error of the inducing algorithm over a hyperparameter search space of considered candidate configurations, usually by evaluating predictions on an independent test set, or by running a resampling scheme such as cross-validation (Bischl et al”