PMLB: a large benchmark suite for machine learning evaluation and comparison

By Randal S. Olson, William La Cava, Patryk Orzechowski, Ryan J. Urbanowicz, Jason H. Moore. 2017

459 citations recorded by the source corpus.

The selection, development, or comparison of machine learning methods in data mining can be a difficult task based on the target problem and goals of a particular study. Numerous publicly available real-world and simulated benchmark datasets have emerged from different sources, but their organization and adoption as standards have been inconsistent. As such, selecting and curating specific benchmarks remains an unnecessary burden on machine learning practitioners and data scientists. The present study introduces an accessible, curated, and developing public benchmark resource to facilitate identification of the strengths and weaknesses of different machine learning methodologies. We compare meta-features among the current set of benchmark datasets in this resource to characterize the diversity of available data. Finally, we apply a number of established machine learning methods to the entire benchmark suite and analyze how datasets and algorithms cluster in terms of performance. From this study, we find that existing benchmarks lack the diversity to properly benchmark machine learning algorithms, and there are several gaps in benchmarking problems that still need to be considered. This work represents another important step towards understanding the limitations of popular benchmarking suites and developing a resource that connects existing benchmarking standards to more diverse and efficient standards in the future.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Learning Optimal Decision Trees with SAT (2018, methodology): “We start with toy benchmarks from [Bessiere et al. , 2009; Olson et al. , 2017] that admit a small DT (less than 15 nodes).”
TPOT: A Tree-based Pipeline Optimization Tool for Automating Machine Learning (2016, background): “Thus, this benchmark—called the Penn Machine Learning Benchmark (PMLB) [14]—represents a comprehensive suite of tests with which to evaluate automated machine learning systems.”
SAT-based Decision Tree Learning for Large Data Sets (2021, methodology): “Additionally, we take the instances from Narodyt-ska et al. (2018) 3 , which were already used before (Bessiere, Hebrard and O’Sullivan 2009; Olson et al. 2017).”
Re-evaluating evaluation (2018, background): “A comprehensive study of performance metrics for machine learning and AI can be found in [31–35].”
Contemporary Symbolic Regression Methods and their Relative Performance (2021, methodology): “To achieve this, we incorporated 130 datasets with ground truth forms into the Penn Machine Learning Benchmark (PMLB) [13], including metadata describing the underlying equations, their units, and various summary statistics.”
FLAML: A Fast and Lightweight AutoML Library (2019, methodology): “We perform extensive evaluation using a recent open source AutoML benchmark (Gijsbers et al., 2019) plus regression datasets from a regression benchmark (Olson et al., 2017).”
OpenML Benchmarking Suites (2017, methodology): “Kaggle.com and PMLB [Olson et al., 2017] offer a Python API for downloading datasets, skdata [Bergstra et al., 2015] offers a Python API for downloading computer vision and natural language processing datasets, and KEEL [Alcala et al., 2010] offers a Java and R API for imbalanced classiﬁcation and…”
The Tree Ensemble Layer: Differentiability meets Conditional Computation (2020, methodology): “23 of these are from the Penn Machine Learning Benchmarks (PMLB) (Olson et al., 2017), and the 3 remaining are CIFAR-10 (Krizhevsky et al.”
A SAT-Based Approach to Learn Explainable Decision Sets (2018, methodology): “The experiments were performed on a subset of datasets of the PMLB repository4 [31].”
PDEBENCH: An Extensive Benchmark for Scientific Machine Learning (2022, methodology): “Generic classiﬁcation problems use, for example, the Penn Machine Learning Benchmark [43].”
OpenML Benchmarking Suites and the OpenML100 (2017, unknown): “Likewise, PMLB (Olson et al., 2017) is another collection of datasets, with strong overlap to UCI, with tools to import them into Python scripts.”
AlphaD3M: Machine Learning Pipeline Synthesis (2021, methodology): “Figure 4 compares performance between different AutoML methods: Autosklearn, TPOT, and Autostacker, and our method AlphaD3M, for a number of common OpenML datasets, which serve as representative benchmark datasets for AutoML systems (Olson et al., 2017; Olson and Moore, 2016; Chen et al., 2018).”
SAT-Based Encodings for Optimal Decision Trees with Explicit Paths (2020, methodology): “These benchmarks were originally obtained by sampling a large set of instances [29], with sampling percentages of 20%, and 50% (we have used the same exact sampled benchmarks as Narodytska et al.).”
Relief-Based Feature Selection: Introduction and Review (2017, background): “However, as similarly argued by Robnik-ˇSikonja and Kononenko (2003), Belanche and Gonz´alez (2011), Bol´on-Canedo et al. (2013), and Olson et al. (2017): to properly evaluate and compare methodologies, diverse simulation studies should ﬁrst be designed and applied.”
Autostacker: a compositional evolutionary learning system (2018, methodology): “To demonstrate the performance of the system described in the preceding sections, 15 datasets were selected from the benchmark dataset [20] —comprising of public data resources, such as OpenML [32] and UCI [16], among others—and served as the sample experimental data for testing Autostacker.”
Interpretable scientific discovery with symbolic regression: a review (2022, unknown): “They can be assessed from the following repositories: the PMLB [55], the OpenML [56], and the UCI [57].”
AutoML to Date and Beyond: Challenges and Opportunities (2020, methodology): “Olson et al. [48] introduced a publicly available dataset suite named Penn Machine Learning Benchmark (PMLB) that initially had 165 publicly available datasets for supervised classification tasks.”