Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data

By Sergei Popov, Stanislav Morozov, Artem Babenko. International Conference on Learning Representations, 2019

433 citations recorded by the source corpus.

Nowadays, deep neural networks (DNNs) have become the main instrument for machine learning tasks within a wide range of domains, including vision, NLP, and speech. Meanwhile, in an important case of heterogenous tabular data, the advantage of DNNs over shallow counterparts remains questionable. In particular, there is no sufficient evidence that deep learning machinery allows constructing methods that outperform gradient boosting decision trees (GBDT), which are often the top choice for tabular problems. In this paper, we introduce Neural Oblivious Decision Ensembles (NODE), a new deep learning architecture, designed to work with any tabular data. In a nutshell, the proposed NODE architecture generalizes ensembles of oblivious decision trees, but benefits from both end-to-end gradient-based optimization and the power of multi-layer hierarchical representation learning. With an extensive experimental comparison to the leading GBDT packages on a large number of tabular datasets, we demonstrate the advantage of the proposed NODE architecture, which outperforms the competitors on most of the tasks. We open-source the PyTorch implementation of NODE and believe that it will become a universal framework for machine learning on tabular data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0b22dbd48ce4e13bdbf0c9d5e86a9cefdaf6d40a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Binary Trees via Sparse Relaxation (2020, methodology): “(13) and all neural network methods (DNDT, NDF, and NODE) using the Quasi-Hyperbolic Adam (Ma & Yarats, 2018) stochastic gradient descent method, with default parameters and batch size equal to 512 .”
Why do tree-based models still outperform deep learning on typical tabular data? (2022, positive): “…to benefit from the flexibility of neural networks while keeping the inductive biases of other algorithms like tree-based models [Lay et al., 2018, Popov et al., 2020, Abutbul et al., 2020, Hehn et al., 2019, Tanno et al., 2019, Chen, 2020, Kontschieder et al., 2015, Rodriguez et al., 2019, Popov…”
Why do tree-based models still outperform deep learning on tabular data? (2022, neutral): “…Popov et al., 2020, Abutbul et al., 2020, Hehn et al., 2019, Tanno et al., 2019, Chen, 2020, Kontschieder et al., 2015, Rodriguez et al., 2019, Popov et al., 2020, Lay et al., 2018] or Factorization Machines Guo et al. [2017], tabular-speciﬁc transformers architectures Somepalli et al.…”
When Do Neural Nets Outperform Boosted Trees on Tabular Data? (2023, methodology): “Despite recent advances in neural network (NN) architectures for tabular data [4, 47], there is still an active debate over whether or not NNs generally outperform gradient-boosted decision trees (GBDTs) on tabular data, with multiple works arguing either for [4, 37, 40, 47, 51] or against [7, 26,…”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “Multiple deep learning methods and architectures have been proposed for tabular datasets, from making discrete decision trees more differentiable [153, 154], regularizing neural networks’ weights [155, 156], to recent attempts using attention-based modules [157, 158, 159].”
Hopular: Modern Hopfield Networks for Tabular Data (2022, background): “Neural Oblivious Decision Ensembles (NODE) generalizes ensembles of oblivious decision trees by hierarchical representation learning (Popov et al., 2019).”
TabLLM: Few-shot Classification of Tabular Data with Large Language Models (2022, positive): “Additional efforts have included differentiable trees, which combine advantages of tree ensembles with gradient based optimization of neural networks (Kontschieder et al., 2015; Popov et al., 2020).”
Tabular Data: Deep Learning is Not All You Need (2021, methodology): “…studies have applied deep learning to the tabular data domain, introducing new neural architectures to achieve improved performance on tabular data [Arik and Pﬁster, 2021, Badirli et al., 2020, Hazimeh et al., 2020, Huang et al., 2020, Klambauer et al., 2017, Baosenguo, 2021, Popov et al., 2020].”
On Embeddings for Numerical Features in Tabular Deep Learning (2022, neutral): “Despite a large number of proposed architectures for tabular DL [2, 3, 13, 17, 21, 24, 31, 39, 40], the performance gap between them and the “shallow” ensembles of decision trees, like GBDT, often remains significant [13, 36].”