OpenML: networked science in machine learning

By Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, Luís Torgo. 2014

1,574 citations recorded by the source corpus.

Many sciences have made significant breakthroughs by adopting online tools that help organize, structure and mine information that is too detailed to be printed in journals. In this paper, we introduce OpenML, a place for machine learning researchers to share and organize data in fine detail, so that they can work more effectively, be more visible, and collaborate with others to tackle harder problems. We discuss how OpenML relates to other examples of networked science and what benefits it brings for machine learning research, individual scientists, as well as students and practitioners.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, ML Systems & Hardware, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

In ML Systems & Hardware

In Signal Processing

Citations in the corpus

BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “We optimized six hyperparameters that control the training procedure (initial learning rate, batch size, dropout, exponential decay factor for learning rate) and the architecture (number of layers, units per layer) of a feed forward neural network for six different datasets gathered from OpenML (Van”
Property Inference Attacks on Fully Connected Neural Networks using Permutation Invariant Representations (2018, background): “There are many online platforms [8, 14, 45] where models are shared openly, including their parameters, thereby providing white-box access.”
Visualizing the Feature Importance for Black Box Models (2018, methodology): “mportance of the interaction is fairly distributed among X 1 and X 2. 6.2 Application on Real Data We demonstrate our graphical tools on the Boston housing data, which is publicly available on OpenML [23] with data set ID 531. The data set contains 13 features that may aect the median home price of ”
Automated Machine Learning: State-of-The-Art and Open Challenges (2019, background): “In addition, openML[124] is a well-organized data repository that allow users to find different datasets, data analysis flows, explore results of these flows, and create tasks that could be shared with the data scientists community and ease the pro-”
Benchmark and Survey of Automated Machine Learning Frameworks (2019, methodology): “More specifically, a combination of the curated benchmarking suites OpenML1006 (Bischl et al., 2017), OpenML-CC187 (Bischl et al., 2019) and AutoML Benchmark8 (Gijsbers et al., 2019) is used.”
Survey on Automated Machine Learning (2019, methodology): “At first, all previously mentioned CASH algorithms are tested on the complete OpenML100 suite.”
Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets (2016, background): “Convolutional neural networks (CNNs) have shown superior performance on a variety of computer vision and speech recognition benchmarks, but ﬁnding good hyperparameter settings is still a challenge, and almost no theoretical guarantees exists.”
Do Not Trust Additive Explanations (2019, methodology): “Numbers on the y-axis are tasks from the OpenML data base ( https://www.openml.org/ ) (Vanschoren et al., 2013), each task corresponds to a different data set.”
Meta-Learning: A Survey (2018, neutral): “∪ P is usually gathered beforehand, or extracted from meta-data repositories [173, 176].”
Automated versus Do-It-Yourself Methods for Causal Inference: Lessons Learned from a Data Analysis Competition (2017, background): “Indeed, there have been several prominent examples of crowdsourcing scientific tools (Vanschoren et al., 2014; Ranard et al., 2014; Paulhamus et al., 2012).”
EvalAI: Towards Better Evaluation Systems for AI Agents (2019, background): “OpenML [34] is an online platform where researchers can automatically log and share machine learning data sets, code, and experiments.”
A comparative evaluation of outlier detection algorithms: Experiments and analyses (2018, background): “OpenML: Networked science in machine learning.”
Hyperparameter Search Space Pruning - A New Component for Sequential Model-Based Hyperparameter Optimization (2015, methodology): “Additionally, we created a new meta-data set which is the largest to the best of our knowledge with about four times more experiments than OpenML and make it publicly available.”
Towards efﬁcient Bayesian Optimization for Big Data (2015, methodology): “We used ﬁve different benchmark datasets (between 1000 and 5000 data points) obtained from [14].”
Revisiting Deep Learning Models for Tabular Data (2021, methodology): “(2014); we use the version with 98K samples available at the OpenML repository (Vanschoren et al., 2014)), ALOI (AL, images, Geusebroek et al.”
Fast Bayesian hyperparameter optimization on large datasets (2017, methodology): “For a more realistic scenario, we optimized the same SVM hyperparameters (see Table 1) without a surrogate on MNIST and two other prominent UCI datasets (gathered from OpenML (Vanschoren et al., 2014)), vehicle registration (Siebert, 1987) and forest cover types (Blackard and Dean, 1999) with more t”
mlr: Machine Learning in R (2016, methodology): “Major directions for future extensions include better support for large-scale data, a closer connection to the OpenML project (Vanschoren et al., 2013) for open machine learning experiments, 1 and better integration of sequential model-based optimization.”
Hyperparameter Importance Across Datasets (2017, methodology): “ functional ANOVA) and should contain close-to-optimal configurations (to allow the construction of good priors). We note that for many common algorithms, the open machine learning environment OpenML [43] already contains very comprehensive performance data for different hyperparameter configuration”
Multi-Objective Counterfactual Explanations (2020, methodology): “re, we let iterated F-racing decide whether our proposed strategies for initialization and mutation of Section 4.3 are preferable. Tuning is performed on six binary classication datasets from OpenML [34] { which were not used in the benchmark. A summary of the tuning setup and results can be found i”