End-to-End Learning of Decision Trees and Forests

By Thomas M. Hehn, Julian F. P. Kooij, Fred A. Hamprecht. 2019

66 citations recorded by the source corpus.

Conventional decision trees have a number of favorable properties, including a small computational footprint, interpretability, and the ability to learn from little training data. However, they lack a key quality that has helped fuel the deep learning revolution: that of being end-to-end trainable. Kontschieder et al. (ICCV, 2015) have addressed this deficit, but at the cost of losing a main attractive trait of decision trees: the fact that each sample is routed along a small subset of tree nodes only. We here present an end-to-end learning scheme for deterministic decision trees and decision forests. Thanks to a new model and expectation–maximization training scheme, the trees are fully probabilistic at train time, but after an annealing process become deterministic at test time. In experiments we explore the effect of annealing visually and quantitatively, and find that our method performs on par or superior to standard learning algorithms for oblique decision trees and forests. We further demonstrate on image datasets that our approach can learn more complex split functions than common oblique ones, and facilitates interpretability through spatial regularization.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Optimization & Training

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, methodology): “Different to those networks using neural units only in routing nodes [101], [102], or routing each sample to predesigned sub-networks [103], [104], adaptive neural tree (ANT) [105] adopts CNN modules as feature transformers in a hard neural tree (see lines with arrows in Fig.”
Neural Prototype Trees for Interpretable Fine-grained Image Recognition (2020, background): “In contrast to most SDTs that require a ﬁxed tree structure, including ours, ANTs greedily learn a binary tree topology.”
MurTree: Optimal Decision Trees via Dynamic Programming and Search (2020, methodology): “…decision trees (Kontschieder et al. (2015); Tanno et al. (2019)), randomised trees (Blanquero et al. (2020)), end-to-end learning of decision trees (Hehn et al. (2019); Elmachtoub et al. (2020)), and dynamic programming methods to construct decision trees from random forests (Vidal and Schiﬀer…”
The Tree Ensemble Layer: Differentiability meets Conditional Computation (2020, methodology): “(Hehn et al., 2019) extended (Kontschieder et al., 2015)’s approach to allow for conditional inference and growing trees level-by-level.”
Why do tree-based models still outperform deep learning on tabular data? (2022, unknown): “…of NNs while keeping the inductive biases of other algorithms like tree-based models [Lay et al., 2018, Popov et al., 2020, Abutbul et al., 2020, Hehn et al., 2019, Tanno et al., 2019, Chen, 2020, Kontschieder et al., 2015, Rodriguez et al., 2019, Popov et al., 2020, Lay et al., 2018] or…”
SegNBDT: Visual Decision Rules for Segmentation (2020, methodology): “While some [11, 22, 7] have tried to fuse decision-tree and deep-learning methods, the resulting models sacrifice both accuracy and interpretability.”
Why do tree-based models still outperform deep learning on typical tabular data? (2022, unknown): “…networks while keeping the inductive biases of other algorithms like tree-based models [Lay et al., 2018, Popov et al., 2020, Abutbul et al., 2020, Hehn et al., 2019, Tanno et al., 2019, Chen, 2020, Kontschieder et al., 2015, Rodriguez et al., 2019, Popov et al., 2020, Lay et al., 2018] or…”
Recent advances in decision trees: an updated survey (2022, methodology): “…but with the recent interest in explainable AI, an increasing number of works also report being inspired by the potential of a hybrid model that combines DT’s interpretability with ANN’s accuracy (Frosst and Hinton 2017; Yang et al. 2018; Silva et al. 2020; Hehn et al. 2020; Wan et al. 2020).”