BranchyNet: Fast inference via early exiting from deep neural networks

By Surat Teerapittayanon, Bradley McDanel, H. T. Kung. 2016

1,542 citations recorded by the source corpus.

Deep neural networks are state of the art methods for many learning tasks due to their ability to extract increasingly better features at each network layer. However, the improved performance of additional layers in a deep network comes at the cost of added latency and energy usage in feedforward inference. As networks continue to get deeper and larger, these costs become more prohibitive for real-time and energy-sensitive applications. To address this issue, we present BranchyNet, a novel deep network architecture that is augmented with additional side branch classifiers. The architecture allows prediction results for a large portion of test samples to exit the network early via these branches when samples can already be inferred with high confidence. BranchyNet exploits the observation that features learned at an early layer of a network may often be sufficient for the classification of many data points. For more difficult samples, which are expected less frequently, BranchyNet will use further or all network layers to provide the best likelihood of correct prediction. We study the BranchyNet architecture using several well-known networks (LeNet, AlexNet, ResNet) and datasets (MNIST, CIFAR10) and show that it can both improve accuracy and significantly reduce the inference time of the network.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Consistent Accelerated Inference via Confident Adaptive Transformers (2021, background): “Adaptive approaches adjust the amount of computation per example to amortize the total inference cost (see Teerapittayanon et al., 2017; Graves, 2017; Huang et al., 2018; Kaya et al., 2019; Wang et al., 2018, inter alia ).”
Dynamic Neural Networks: A Survey (2021, background): “early exiting is typically achieved according to confidencebased criteria [45], [51] or learned functions [46], [52], [53].”
Why Should We Add Early Exits to Neural Networks? (2020, background): “)] is always normalized between 0 and 1. By comparing H [ci(x)] against a user-selectedthresholdi, wecanmaketheﬁnaldecisiononwhethertoexitthearchitecture. This strategy was popularized by BranchyNet [68,69]. The drawback of this approach is that a separate threshold must be selected for each auxilia”
Depth-Adaptive Transformer (2019, methodology): “lassiﬁcation work of Huang et al. (2017) and Bolukbasi et al. (2017) to structured prediction where dynamic computation decisions impact future computation. Related work from computer vision includes Teerapittayanon et al. (2016); Figurnov et al. (2017) and Wang et al. (2018) who explored the idea o”
SkipNet: Learning Dynamic Routing in Convolutional Networks (2017, background): “[5] and [29] propose the use of early termination in convolutional networks.”
FastBERT: a Self-distilling BERT with Adaptive Inference Time (2020, background): “…Conventional approaches in adaptive computations are performed token-wise or patch-wise, who either adds recurrent steps to individual tokens (Graves, 2016) or dynamically adjusts the number of executed layers inside discrete regions of images (Teerapittayanon et al., 2016; Figurnov et al., 2017).”
Pre-trained models for natural language processing: A survey (2020, methodology): “The idea of early exit is first applied in computer vision, such as BranchyNet [182] and Shallow-Deep Network [183].”
Spatially Adaptive Computation Time for Residual Networks (2016, methodology): “BranchyNet [40] is the most similar approach to ours although only applicable to classification problems.”
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference (2020, positive): “Our work is inspired by Cambazoglu et al. (2010), Teerapittayanon et al. (2017), and Huang et al. (2018), but mainly differs from previous work in that we focus on improving model efﬁciency with minimal quality degradation.”
Convolutional Networks with Adaptive Inference Graphs (2017, methodology): “To adapt computation time in convolutional networks, (Huang et al. 2017; Teerapittayanon et al. 2016) propose architectures that add classification branches to intermediate layers.”
BlockDrop: Dynamic Inference Paths in Residual Networks (2017, background): “[48] propose BranchyNet, a network composed of branches at each layer to make early classification decisions.”
Salient Object Detection in the Deep Learning Era: An In-Depth Survey (2019, background): “BranchyNet [148] stops the computation early once the classiﬁcation entropy of the added intermediate classiﬁcation branches falls below a threshold.”
MSD: Multi-Self-Distillation Learning via Multi-classifiers within Deep Neural Networks (2019, background): “ed platforms (e.g., mobile phones) or applications requiring a short response time (e.g., self-driving cars). To lessen these increasing costs, multi-exit network architectures Larsson et al. (2016); Teerapittayanon et al. (2016) are proposed to lessen the inference costs by allowing prediction to e”
Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing (2019, methodology): “In this paper, we implement the branchy model with the assist of the open-source BranchyNet [40] framework and Chainer [41] framework.”
Edge Intelligence: On-Demand Deep Learning Model Co-Inference with Device-Edge Synergy (2018, background): “Currently, the early-exit mechanism has been supported by the open source framework BranchyNet[15].”
A Dynamic Deep Neural Network Design for Efficient Workload Allocation in Edge Computing (2017, methodology): “It is observed that under the same overall accuracy requirements, our fine-grained AO criteria is able to report more classification results than CDL and BranchyNet.”
Hard-Aware Deeply Cascaded Embedding (2016, background): “BranchyNet [34] attempts to speed up image classification by taking advantage of the DSN framework during test phase, where an image will be predicted using features learned at an early layer if high confidence score can be achieved.”