Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science

By Randal S. Olson, Nathan Bartley, Ryan J. Urbanowicz, Jason H. Moore. 2016

620 citations recorded by the source corpus.

As the field of data science continues to grow, there will be an ever-increasing demand for tools that make machine learning accessible to non-experts. In this paper, we introduce the concept of tree-based pipeline optimization for automating one of the most tedious parts of machine learning--pipeline design. We implement an open source Tree-based Pipeline Optimization Tool (TPOT) in Python and demonstrate its effectiveness on a series of simulated and real-world benchmark data sets. In particular, we show that TPOT can design machine learning pipelines that provide a significant improvement over a basic machine learning analysis while requiring little to no input nor prior knowledge from the user. We also address the tendency for TPOT to design overly complex pipelines by integrating Pareto optimization, which produces compact pipelines without sacrificing classification accuracy. As such, this work represents an important step toward fully automating machine learning pipeline design.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Benchmark and Survey of Automated Machine Learning Frameworks (2019, background): “For example, genetic programming could create a pipeline for a classification task without any classification algorithm (Olson et al., 2016).”
Survey on Automated Machine Learning (2019, methodology): “For example TPOT could create a pipeline for a classification task without any classification algorithm (Olson et al., 2016a).”
Auto-Keras: An Efficient Neural Architecture Search System (2018, background): “Similar to SMAC [15], TPOT [28], Auto-WEKA [35], and Auto-Sklearn [13], the goal is to enable domain experts who are not familiar with machine learning technologies to use machine learning techniques easily.”
Meta-Learning: A Survey (2018, neutral): “Other approaches give recommendations for certain pipeline steps [118, 162], and can be leveraged in larger pipeline construction approaches, such as planning [105, 74, 55, 183] or evolutionary techniques [110, 163].”
Automated Machine Learning: From Principles to Practices (2018, methodology): “…learning configurations, the iterative differentiation methods first iteratively update the model parameter for n steps to obtain 𝑤 𝑛 ( 𝛼 ) for the optimization of lower level objective, and then uses the updated result 𝑤 𝑛 ( 𝛼 ) to approximate ∗ ( 𝛼 ) to obtain the approximated gradient.”
Taking Human out of Learning Applications: A Survey on Automated Machine Learning (2018, methodology): “For example, evolutionary algorithms has been applied in feature selection and generation [50], [86], [87], [88], [89], and model selection [90].”
FeatureHub: Towards Collaborative Data Science (2017, methodology): “In a similar approach, [16] use genetic programming to optimize”
One button machine for automating feature engineering in relational databases (2017, neutral): “TPOT [10] is another system that uses genetic programming to ﬁnd the best model conﬁguration and preprocessing work-ﬂow.”
AutoML to Date and Beyond: Challenges and Opportunities (2020, methodology): “• TPOT [U39] is a tree-based optimization tool to automate machine learning pipelines developed by Olson et al. [47].”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, methodology): “Consequently, they might not deploy an AutoML model despite all performance gains.”
Automated Selection and Configuration of Multi-Label Classification Algorithms with Grammar-Based Genetic Programming (2018, methodology): “Most Auto-ML systems proposed to date focus on generating sequences of steps to solve single label classiﬁcation (SLC) problems [3, 5, 6, 8, 15].”
Evolutionary Generative Adversarial Networks (2018, background): “Last but not the least, the evolutionary learning paradigm has been utilized to solve a number of deep/machine tasks, such as automatic machine learning [62], multiobjective optimization [63], etc.”
SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization (2021, methodology): “Besides simple random search (Bergstra and Bengio, 2012), evolutionary algorithms (Olson et al., 2016) and bandit approaches (Li et al., 2018), Bayesian Optimization (BO) (Mockus et al., 1978; Shahriari et al., 2016) is a commonly used approach for Hyperparameter Optimization (HPO) (Feurer and…”
RECIPE: A Grammar-Based Framework for Automatically Evolving Classification Pipelines (2017, background): “For example, TPOT can create a pipeline without a classification algorithm [3].”
Autostacker: a compositional evolutionary learning system (2018, background): “Recent efforts in AutoML, such as AutoSklearn[7] and TPOT[19], demonstrate success when applied on a variety of datasets.”