COCO: a platform for comparing continuous optimizers in a black-box setting

By Nikolaus Hansen, Anne Auger, Olaf Mersmann, Tea Tusar, Dimo Brockhoff. Optim. Methods Softw., 2016

546 citations recorded by the source corpus.

We introduce COCO, an open-source platform for Comparing Continuous Optimizers in a black-box setting. COCO aims at automatizing the tedious and repetitive task of benchmarking numerical optimization algorithms to the greatest possible extent. The platform and the underlying methodology allow to benchmark in the same framework deterministic and stochastic solvers for both single and multiobjective optimization. We present the rationals behind the (decade-long) development of the platform as a general proposition for guidelines towards better benchmarking. We detail underlying fundamental concepts of COCO such as the definition of a problem as a function instance, the underlying idea of instances, the use of target values, and runtime defined by the number of function calls as the central performance measure. Finally, we give a quick overview of the basic code structure and the currently available test suites.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Numerical Analysis

View Paper: https://www.semanticscholar.org/paper/c20191b2849d10e5861bf6c6a2c311e0343f1417

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Numerical Analysis

Citations in the corpus

OpenML Benchmarking Suites (2017, methodology): “…environments such as the OpenAI Gym [Brockman et al., 2016] run and evaluate reinforcement learning experiments, the COCO suite standardizes benchmarking for black-box optimization [Hansen et al., 2020] and ASLib provides a benchmarking protocol for algorithm selection [Bischl et al., 2016a].”
On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice (2020, methodology): “For example, there is a platform called COCO (Comparing Continuous Optimizers) [127] that provides benchmarks and analyzes common continuous optimizers.”
Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020 (2021, background): “There have been benchmarks and competitions on compare black-box optimization such as the annual COCO competition [28].”
Bio-inspired computation: Where we stand and what's next (2019, background): “BBOB [327] Functions with increasing dimensionality (from small dimensions)”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, methodology): “A benchmarking suite for general continuous black box optimization is COCO (Hansen et al., 2021), which contains a well-curated collection of various synthetic benchmark functions and has been used in the general black box optimization community for many years.”
Automated Algorithm Selection: Survey and Perspectives (2018, unknown): “A total of 480 BBOB instances (instance IDs 1–5 of all 24 BBOB functions, across dimensions 2, 3, 5 and 10) were considered, combined with respective results of all 129 solvers submitted to the COCO platform (Hansen et al., 2016) so far.”