Deep Neural Decision Trees

By Yongxin Yang, Irene Garcia Morillo, Timothy M. Hospedales. 2018

199 citations recorded by the source corpus.

Deep neural networks have been proven powerful at processing perceptual data, such as images and audio. However for tabular data, tree-based models are more popular. A nice property of tree-based models is their natural interpretability. In this work, we present Deep Neural Decision Trees (DNDT) -- tree models realised by neural networks. A DNDT is intrinsically interpretable, as it is a tree. Yet as it is also a neural network (NN), it can be easily implemented in NN toolkits, and trained with gradient descent rather than greedy splitting. We evaluate DNDT on several tabular datasets, verify its efficacy, and investigate similarities and differences between DNDT and vanilla decision trees. Interestingly, DNDT self-prunes at both split and feature-level.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, methodology): “Neural networks like neural additive models [5] and neural decision trees [322] could potentially be used to model c → y, since they are both differentiable, nonlinear, and intrinsically interpretable once the input features are interpretable.”
Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses From a Practical Point of View (2019, background): “Consequently, new models fusing white- and black-box approaches are necessary [150]–[152].”
Mathematical optimization in classification and regression trees (2021, background): “In Optimal Randomized Classification and Regression Trees, the splitting rule at branch node t ∈ B is probabilistic (Irsoy et al. 2012; Yang et al. 2018), i.e., individuals move with a certain probability to the left child of t and with the complementary probability to the right one.”
Recent advances in decision trees: an updated survey (2022, background): “…but with the recent interest in explainable AI, an increasing number of works also report being inspired by the potential of a hybrid model that combines DT’s interpretability with ANN’s accuracy (Frosst and Hinton 2017; Yang et al. 2018; Silva et al. 2020; Hehn et al. 2020; Wan et al. 2020).”
Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data (2019, methodology): “While the importance of deep learning on tabular data is recognized by the ML community, and many works address this problem (Zhou & Feng, 2017; Yang et al., 2018; Miller et al., 2017; Lay et al., 2018; Feng et al., 2018; Ke et al., 2018), the proposed DNN approaches do not consistently outperform t”
FOCUS: Flexible Optimizable Counterfactual Explanations for Tree Ensembles (2019, unknown): “Neural decision trees (Balestriero 2017; Yang, Morillo, and Hospedales 2018) are also differentiable versions of trees, which use a full neural network instead of a simple sigmoid.”
A Survey on the Explainability of Supervised Machine Learning (2020, methodology): “Yang et al. (2018a) propose a binary decision tree that represents the most important decision rules.”
TabNet: Attentive Interpretable Tabular Learning (2019, background): “Yet, conventional DNNs, DTs, and even their hybrid variant of deep neural DTs (Yang, Morillo, and Hospedales 2018) severely suffer from the imbalanced data and cannot learn the required sorting and ranking operations (Yang, Morillo, and Hospedales 2018).”
TabTransformer: Tabular Data Modeling Using Contextual Embeddings (2020, methodology): “…for tabular data, there are deep versions of factorization machines (Guo et al. 2018; Xiao et al. 2017), Transformers-based methods (Song et al. 2019; Li et al. 2020; Sun et al. 2019), and deep versions of decision-tree-based algorithms (Ke et al. 2019; Yang, Morillo, and Hospedales 2018).”
Learning Binary Trees via Sparse Relaxation (2020, methodology): “We also consider two baselines with probabilistic routing: deep neural decision trees (DNDT; Yang et al., 2018) and deep neural decision forests (Kontschieder et al., 2015) conﬁgured to use an ensemble size of 1 (NDF-1).”
Revisiting Deep Learning Models for Tabular Data (2021, background): “To address this issue, several works (Hazimeh et al., 2020; Kontschieder et al., 2015; Popov et al., 2020; Yang et al., 2018) propose to “smooth” decision functions in the internal tree nodes to make the overall tree function and tree routing differentiable.”