Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation

By Rasool Fakoor, Jonas W. Mueller, Nick Erickson, Pratik Chaudhari, Alex Smola. 2020

63 citations recorded by the source corpus.

Automated machine learning (AutoML) can produce complex model ensembles by stacking, bagging, and boosting many individual models like trees, deep networks, and nearest neighbor estimators. While highly accurate, the resulting predictors are large, slow, and opaque as compared to their constituents. To improve the deployment of AutoML on tabular data, we propose FAST-DAD to distill arbitrarily complex ensemble predictors into individual models like boosted trees, random forests, and deep networks. At the heart of our approach is a data augmentation strategy based on Gibbs sampling from a self-attention pseudolikelihood estimator. Across 30 datasets spanning regression and binary/multiclass classification tasks, FAST-DAD distillation produces significantly better individual models than one obtains through standard training on the original data. Our individual distilled models are over 10x faster and more accurate than ensemble predictors produced by AutoML tools like H2O/AutoSklearn.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, background): “Ensemble models on the other hand, have been shown to be promising teachers from the early work of [9] until recent works in various domains [10, 68, 54, 56] and with techniques to boost the their performance [52, 17, 41].”
Knowledge Distillation: A Survey (2020, background): “…deployed to various excellent learning schemes, such as the adversarial learning (Liu et al. 2018), auto machine learning (Macko et al. 2019; Fakoor et al. 2020), label noise ﬁltering learning (Xia et al. 2018), lifelong learning (Zhai et al. 2019), and reinforcement learning (Ashok et al.…”
Does Knowledge Distillation Really Work? (2021, methodology): “[12] developed a data-augmentation scheme for the distillation of large ensembles of simple models for tabular data, achieving impressive results on a wide range of tabular benchmarks.”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, background): “Recently, model distillation approaches have been proposed to extract a simple(r) model from a complex ensemble (Fakoor et al., 2020).”