The Tree Ensemble Layer: Differentiability meets Conditional Computation

By Hussein Hazimeh, Natalia Ponomareva, P. Mol, Zhenyu Tan, Rahul Mazumder. 2020

97 citations recorded by the source corpus.

Neural networks and tree ensembles are state-of-the-art learners, each with its unique statistical and computational advantages. We aim to combine these advantages by introducing a new layer for neural networks, composed of an ensemble of differentiable decision trees (a.k.a. soft trees). While differentiable trees demonstrate promising results in the literature, they are typically slow in training and inference as they do not support conditional computation. We mitigate this issue by introducing a new sparse activation function for sample routing, and implement true conditional computation by developing specialized forward and backward propagation algorithms that exploit sparsity. Our efficient algorithms pave the way for jointly training over deep and wide tree ensembles using first-order methods (e.g., SGD). Experiments on 23 classification datasets indicate over 10x speed-ups compared to the differentiable trees used in the literature and over 20x reduction in the number of parameters compared to gradient boosted trees, while maintaining competitive performance. Moreover, experiments on CIFAR, MNIST, and Fashion MNIST indicate that replacing dense layers in CNNs with our tree layer reduces the test loss by 7-53% and the number of parameters by 8x. We provide an open-source TensorFlow implementation with a Keras API.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Optimization & Training

Citations in the corpus

DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning (2021, neutral): “This function has been recently used for conditional computation in soft trees [11] and is popular in the literature on computer graphics [8, 30].”
Dynamic Neural Networks: A Survey (2021, background): “in routing nodes [96], [97], or routing each sample to pre-designed sub-networks [98], [99], adaptive neural tree (ANT) [100] adopts CNN modules as feature transformers”
Tabular Data: Deep Learning is Not All You Need (2021, methodology): “Several works address this issue by smoothing the decision functions in the internal tree nodes to make the tree function and tree routing differentiable [Hazimeh et al., 2020, Kontschieder et al., 2015, Popov et al., 2020].”
On Embeddings for Numerical Features in Tabular Deep Learning (2022, unknown): “During several recent years, the community has proposed a large number of deep models for tabular data [2, 3, 13, 15, 17, 21, 24, 31, 39, 40, 46].”
How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy (2023, methodology): “(Hazimeh et al., 2020)) and subsequently trained conventionally using DP training procedures.”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “[153] Hussein Hazimeh, Natalia Ponomareva, Petros Mol, Zhenyu Tan, and Rahul Mazumder.”
Revisiting Deep Learning Models for Tabular Data (2021, background): “For these reasons, a large number of DL solutions were recently proposed, and new models continue to emerge (Arik and Pfister, 2020; Badirli et al., 2020; Hazimeh et al., 2020; Huang et al., 2020a; Klambauer et al., 2017; Popov et al., 2020; Song et al., 2019; Wang et al., 2017, 2020a).”