Testing heuristics: We have it all wrong

By John N. Hooker. 1995

Subfields: Cognitive Science & Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Science & Psychology

Citations in the corpus

AI and the Everything in the Whole Wide World Benchmark (2021, background): “empirical and incremental work rather than hypothesis-based scientific inquiry [Hooker, 1995].”
Learning to Branch in Mixed Integer Programming (2016, background): “The case for focusing on nodes versus time in benchmarking branching methods is discussed in depth in (Hooker 1995).”
Vehicle Routing Problem with Time Windows, Part I: Route Construction and Local Search Algorithms (2005, background): “Even more difficult issues to address are differences in coding skill, tuning, and effort invested (Hooker 1995).”
Metaheuristics - the metaphor exposed (2015, unknown): “In general, the testing of algorithms is done in an ad-hoc way, and no serious (statistical) testing is generally required for a claim of good performance to be accepted (Hooker, 1995).”
Logic-based Benders decomposition (2003, unknown): “This provided a controlled experiment in which the effect of decomposition could be isolated, a general approach advocated in [24, 26].”
A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the CEC’2005 Special Session on Real Parameter Optimization (2009, background): “The work of Hooker is pioneer in this line and it shows an interesting study on what we must do and not do when we suggest the analysis of the behaviour of a metaheuristic about a problem (Hooker 1997).”
A Survey of Automatic Parameter Tuning Methods for Metaheuristics (2020, background): “The demand of systematic approaches for metaheuristics’ parameter setting has been increasingly outlined in the literature since the end of the last century [16], [17].”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “Most obviously, the use of benchmarks to assess progress in the ﬁeld creates a competitive testing dynamic that emphasizes outcomes rather than proper scientiﬁc inquiry [66].”
COCO: a platform for comparing continuous optimizers in a black-box setting (2016, background): “Johnson [51] and Hooker [46] provide excellent discussions of how to do experimentation beyond the specific benchmarking scenario.”
Protecting against evaluation overfitting in empirical reinforcement learning (2011, background): “Others have pointed out statistical problems in typical evaluations [22] or bemoaned the emphasis they create on software engineering instead of research innovation [26].”
Parameter tuning for configuring and analyzing evolutionary algorithms (2011, methodology): “As explained in [1], there is a big difference between competitive and scientific testing of heuristic methods.”
Hybrid metaheuristics in combinatorial optimization: A survey (2011, background): “In fact, among the key points of the engineering process of a hybrid metaheuristic are scientiﬁc testing [163, 164] and the statistical assessment of the results [165].”
jMetal: A Java framework for multi-objective optimization (2011, background): “To start with, it is worth highlighting that metaheuristics are non-deterministic techniques, so trying to draw some conclusion after running once one of them does not make sense [48].”
Automatically improving the anytime behaviour of optimisation algorithms (2014, methodology): “The use of experimental design techniques [2, 18, 35, 36] began a trend in which the task of finding the most promising configurations to be tested is performed automatically.”
The jMetal framework for multi-objective optimization: Design and architecture (2010, methodology): “As metaheuristics are non-deterministic techniques , trying to infer some conclusion after one execution of them does not make sense [45], so the usual procedure is to carry out a numberN of independent runs per experiment and then to use means and standard deviations (or medians and interquartile r”