Neural Decision Trees
By Randall Balestriero. 2017
56 citations recorded by the source corpus.
In this paper we propose a synergistic melting of neural networks and decision trees (DT) we call neural decision trees (NDT). NDT is an architecture a la decision tree where each splitting node is an independent multilayer perceptron allowing oblique decision functions or arbritrary nonlinear decision function if more than one layer is used. This way, each MLP can be seen as a node of the tree. We then show that with the weight sharing asumption among those units, we end up with a Hashing Neural Network (HNN) which is a multilayer perceptron with sigmoid activation function for the last layer as opposed to the standard softmax. The output units then jointly represent the probability to be in a particular region. The proposed framework allows for global optimization as opposed to greedy in DT and differentiability w.r.t. all parameters and the input, allowing easy integration in any learnable pipeline, for example after CNNs for computer vision tasks. We also demonstrate the modeling power of HNN allowing to learn union of disjoint regions for final clustering or classification making it more general and powerful than standard softmax MLP requiring linear separability thus reducing the need on the inner layer to perform complex data transformations. We finally show experiments for supervised, semi-suppervised and unsupervised tasks and compare results with standard DTs and MLPs.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Harrie Oosterhuis
- Nadia Burkart
- Maarten de Rijke
- Marco F. Huber
- Ana Lucic
- Hinda Haned
- W. Bruce Croft
- Gesina Schwalbe
In Computer Vision and Pattern Recognition
Citations in the corpus
- Deep Neural Decision Trees (2018, neutral): “Despite the similar name, our work is fundamentally different to Balestriero (2017) which developed a kind of ‘oblique’ decision tree realised by neural network.”
- FOCUS: Flexible Optimizable Counterfactual Explanations for Tree Ensembles (2019, background): “Neural decision trees [2, 37] are also differentiable versions of trees, which use a full neural network instead of a simple sigmoid.”
- A Survey on the Explainability of Supervised Machine Learning (2020, methodology): “…OT-SpAM Classification Tree-based Wang et al. (2015a) Tree Regularization Classification Tree-based Wu et al. (2018) NDT Classification Tree-based Balestriero (2017) DNDT Classification Tree-based Yang et al. (2018b) LP relaxation Classification Rule-based Malioutov et al. (2017) 1R…”