NBDT: Neural-Backed Decision Trees

By Alvin Wan, Lisa Dunlap, Daniel Ho, Jihan Yin, Scott Lee, Henry Jin, Suzanne Petryk, Sarah Adel Bargal, Joseph E. Gonzalez. 2020

125 citations recorded by the source corpus.

Machine learning applications such as finance and medicine demand accurate and justifiable predictions, barring most deep learning methods from use. In response, previous work combines decision trees with deep learning, yielding models that (1) sacrifice interpretability to maintain accuracy or (2) underperform modern neural networks to maintain interpretability. We forgo this dilemma by proposing Neural-Backed Decision Trees (NBDTs), modified hierarchical classifiers that use trees constructed in weight-space. Our NBDTs achieve (1) interpretability and (2) neural network accuracy: We preserve interpretable properties -- e.g., leaf purity and a non-ensembled model -- and demonstrate interpretability of model predictions both qualitatively and quantitatively. Furthermore, NBDTs match state-of-the-art neural networks on CIFAR10, CIFAR100, TinyImageNet, and ImageNet to within 1-2%. This yields state-of-the-art interpretable models on ImageNet, with NBDTs besting all decision-tree-based methods by ~14% to attain 75.30% top-1 accuracy. Code and pretrained NBDTs are at this https URL.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Neural Prototype Trees for Interpretable Fine-grained Image Recognition (2020, methodology): “Other tree approaches for image classification apply post-hoc explanation techniques, by showing example images per node [2, 55], visualizing typical CNN filters of each node that can be manually labelled [55], showing class activation maps [25] or manual inspection of leaf labels and the meaning of”
SegNBDT: Visual Decision Rules for Segmentation (2020, background): “Interpretable Segmentation Models: Likwise, recent XAI work tackles directly interpretable classification models [32, 18, 23, 1, 4], but analogous work in segmentation is sparse.”
Learning Decision Trees Recurrently Through Communication (2021, background): “NBDT [59] constructs trees in the weight space of a neural network and Adaptive Neural Trees [56] directly model the neural network as a decision tree, where each node and edge correspond to one or more network modules.”
Label-Free Concept Bottleneck Models (2023, background): “NBDT (Wan et al., 2020) propose replacing the final layer with a neural backed decision tree for another form of more interpretable decisions.”
Recent advances in decision trees: an updated survey (2022, methodology): “Most of these contributions are motivated by the sub-optimality argument—greedy algorithms produce sub-optimal trees, therefore the non-greedy gradient-based optimization may help (Suarez and Lut-sko 1999; Irsoy et al. 2012, 2014; Norouzi et al. 2015), but with the recent interest in explainable AI,”