Decision Forests, Convolutional Networks and the Models in-Between

By Yani Ioannou, Duncan P. Robertson, Darko Zikic, Peter Kontschieder, Jamie Shotton, Matthew Brown, Antonio Criminisi. 2016

93 citations recorded by the source corpus.

This paper investigates the connections between two state of the art classifiers: decision forests (DFs, including decision jungles) and convolutional neural networks (CNNs). Decision forests are computationally efficient thanks to their conditional computation property (computation is confined to only a small region of the tree, the nodes along a single branch). CNNs achieve state of the art accuracy, thanks to their representation learning capabilities. We present a systematic analysis of how to fuse conditional computation with representation learning and achieve a continuum of hybrid models with different ratios of accuracy vs. efficiency. We call this new family of hybrid models conditional networks. Conditional networks can be thought of as: i) decision trees augmented with data transformation operators, or ii) CNNs, with block-diagonal sparse weight matrices, and explicit data routing functions. Experimental validation is performed on the common task of image classification on both the CIFAR and Imagenet datasets. Compared to state of the art CNNs, our hybrid models yield the same accuracy with a fraction of the compute cost and much smaller number of parameters.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning (2021, neutral): “Our work is also related to the conditional computation models that activate parts of the neural network based on the input [2, 1, 33, 12, 36].”
Dynamic Neural Networks: A Survey (2021, background): “A subsequent work [104] not only partitions samples to different sub-networks, but also divides and routes the feature channels.”
Adaptive Neural Trees (2018, methodology): “…2017) RDT (L´eon & Denoyer, 2015) BT (˙Irsoy et al., 2014) Conv DT (Laptev & Buhmann, 2014) NDT (Rota Bulo & Kontschieder, 2014) NDT 2 (Xiao, 2017) NDF (Kontschieder et al., 2015) CNet (Ioannou et al., 2016) ANT (ours) layers in the routers to learn more complex partitionings of the input space.”
Deep convolutional autoencoder for radar-based classification of similar aided and unaided human activities (2018, unknown): “By not squashing the data between 0 and 1, ReLU has been shown to prevent the vanishing gradient problem and has the additional advantage of enabling a sparse representation of the data when then network is initialized randomly [38].”
Protein model quality assessment using 3D oriented convolutional neural networks (2018, methodology): “As a second attempt, we later changed our network architecture to let the network learn the data routing as proposed in (26).”
Neural Random Forests (2016, methodology): “The conditional networks from Ioannou et al. (2016) also use trainable routing functions to perform conditional transformations on the inputs—they are thus capable of transferring computational eﬃciency beneﬁts of decision trees into the domain of convolutional networks.”
The Tree Ensemble Layer: Differentiability meets Conditional Computation (2020, methodology): “, 2015) N N Y Soft tree/Alter (Ioannou et al., 2016) N H Y Tree-NN/SGD (Frosst & Hinton, 2017) N H Y Soft tree/SGD (Zoran et al.”
HydraNets: Specialized Dynamic Architectures for Efficient Inference (2018, background): “[20, 17] improve accuracy with significant increase in cost relative to the baseline architecture.”
Deciding How to Decide: Dynamic Routing in Artificial Neural Networks (2017, background): “Additionally, Bengio et al. (2015) explored dynamically skipping layers in neural networks, and Ioannou et al. (2016) explored dynamic routing in networks with equal-length paths.”
End-to-End Learning of Decision Trees and Forests (2019, background): “Various works are now exploring the relation between both classiﬁcation approaches (Ioannou et al. 2016; Wang et al. 2017), such as the Deep Neural Decision Forests (DNDFs) (Kontschieder et al. 2015).”