Adaptive Neural Trees

By Ryutaro Tanno, Kai Arulkumaran, Daniel C. Alexander, Antonio Criminisi, Aditya V. Nori. 2018

181 citations recorded by the source corpus.

Deep neural networks and decision trees operate on largely separate paradigms; typically, the former performs representation learning with pre-specified architectures, while the latter is characterised by learning hierarchies over pre-specified features with data-driven architectures. We unite the two via adaptive neural trees (ANTs) that incorporates representation learning into edges, routing functions and leaf nodes of a decision tree, along with a backpropagation-based training algorithm that adaptively grows the architecture from primitive modules (e.g., convolutional layers). We demonstrate that, whilst achieving competitive performance on classification and regression datasets, ANTs benefit from (i) lightweight inference via conditional computation, (ii) hierarchical separation of features useful to the task e.g. learning meaningful class associations, such as separating natural vs. man-made objects, and (iii) a mechanism to adapt the architecture to the size and complexity of the training dataset.

Subfields: Artificial Intelligence, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “Different to those networks using neural units only in routing nodes [100], [101], or routing each sample to predesigned sub-networks [102], [103], adaptive neural tree (ANT) [104] adopts CNN modules as feature transformers in a hard neural tree (see lines with arrows in Fig.”
MurTree: Optimal Decision Trees via Dynamic Programming and Search (2020, background): “…(2020)), a ﬁne-grained computational complexity study (Ordyniak and Szei-der (2021)), neural networks for decision trees (Kontschieder et al. (2015); Tanno et al. (2019)), randomised trees (Blanquero et al. (2020)), end-to-end learning of decision trees (Hehn et al. (2019); Elmachtoub et al.…”
Conservative Q-Improvement: Reinforcement Learning for an Interpretable Decision-Tree Policy (2019, methodology): “Some work has explored combining neural nets and decision tree forms [Tanno et al. , 2018; Zhao, 2001], including using decision trees to explain neural nets [Zhang et al. , 2018].”
XOC: Explainable Observer-Classifier for Explainable Binary Decisions (2019, background): “Adaptive Neural Trees (Tanno et al., 2018) directly model the neural network as a decision tree, where each node and edge correspond to one or more modules of the network.”
Surprisal-Triggered Conditional Computation with Neural Networks (2020, background): “This often takes the form of a mixture-of-experts [23, 24], in which a learned controller is used to select the experts relevant for a given input [25, 26, 27, 28, 29, 30], possibly hierarchically [31, 32, 33, 34].”
NBDT: Neural-Backed Decision Trees (2020, background): “…papers or improved using a modern backbone: Deep Neural Decision Forest (DNDF updated with ResNet18) [16], Explainable Observer-Classiﬁer (XOC) [2], Deep Convolutional Decision Jungle (DCDJ) [3], Network of Experts (NofE) [1], Deep Decision Network (DDN) [23], and Adaptive Neural Trees (ANT) [33].”
Optimal Decision Trees for Nonlinear Metrics (2020, methodology): “For completeness, we note that there are other works that follow a different but related research line, which use neural networks to learn decision trees/forests (Tanno et al. 2019; Kontschieder et al. 2015), consider more general tree structures (Yang, Shen, and Gao 2019), or end-to-end learning us”
Learning Decision Trees Recurrently Through Communication (2021, background): “NBDT [59] constructs trees in the weight space of a neural network and Adaptive Neural Trees [56] directly model the neural network as a decision tree, where each node and edge correspond to one or more network modules.”
Attention Convolutional Binary Neural Tree for Fine-Grained Visual Categorization (2019, methodology): “It demonstrates the effectiveness of using multiple deep models concentrating on different object regions in FGVC. Inspired by [41], we design an attention convolutional binary neural tree architecture (ACNet) for weakly supervised FGVC.”
Why do tree-based models still outperform deep learning on typical tabular data? (2022, unknown): “…the inductive biases of other algorithms like tree-based models [Lay et al., 2018, Popov et al., 2020, Abutbul et al., 2020, Hehn et al., 2019, Tanno et al., 2019, Chen, 2020, Kontschieder et al., 2015, Rodriguez et al., 2019, Popov et al., 2020, Lay et al., 2018] or Factorization Machines…”
Why do tree-based models still outperform deep learning on tabular data? (2022, unknown): “…keeping the inductive biases of other algorithms like tree-based models [Lay et al., 2018, Popov et al., 2020, Abutbul et al., 2020, Hehn et al., 2019, Tanno et al., 2019, Chen, 2020, Kontschieder et al., 2015, Rodriguez et al., 2019, Popov et al., 2020, Lay et al., 2018] or Factorization Machines…”
TabNet: Attentive Interpretable Tabular Learning (2019, methodology): “(Ke et al. 2019) proposes a DNN architecture by explicitly leveraging expressive feature combinations, however, learning is based on transferring knowledge from gradient-boosted DT. (Tanno et al. 2018) proposes a DNN architecture by adaptively growing from primitive blocks while representation learn”
Recent advances in decision trees: an updated survey (2022, methodology): “Most of these contributions are motivated by the sub-optimality argument—greedy algorithms produce sub-optimal trees, therefore the non-greedy gradient-based optimization may help (Suarez and Lut-sko 1999; Irsoy et al. 2012, 2014; Norouzi et al. 2015), but with the recent interest in explainable AI,”