IDK Cascades: Fast Deep Learning by Learning not to Overthink

By Xin Wang, Yujia Luo, Daniel Crankshaw, Alexey Tumanov, Joseph E. Gonzalez. 2017

136 citations recorded by the source corpus.

Advances in deep learning have led to substantial increases in prediction accuracy but have been accompanied by increases in the cost of rendering predictions. We conjecture that fora majority of real-world inputs, the recent advances in deep learning have created models that effectively "overthink" on simple inputs. In this paper, we revisit the classic question of building model cascades that primarily leverage class asymmetry to reduce cost. We introduce the "I Don't Know"(IDK) prediction cascades framework, a general framework to systematically compose a set of pre-trained models to accelerate inference without a loss in prediction accuracy. We propose two search based methods for constructing cascades as well as a new cost-aware objective within this framework. The proposed IDK cascade framework can be easily adopted in the existing model serving systems without additional model re-training. We evaluate the proposed techniques on a range of benchmarks to demonstrate the effectiveness of the proposed framework.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, methodology): “For example, multi-exit models can be used to: 1) alleviate the over-thinking issue while reducing the overall computation [50], [266]; 2) perform long-tailed classification [267] by inducing early exiting in the training stage; and 3) improve the model robustness [268].”
Why Should We Add Early Exits to Neural Networks? (2020, methodology): “, cascaded networks [45], IDK networks [70], deeply supervised models [36]).”
SkipNet: Learning Dynamic Routing in Convolutional Networks (2017, background): “Related to the work on early termination, several others [1, 9, 21, 33] investigate cascaded model composition.”
Differentiable Branching In Deep Networks for Fast Inference (2020, methodology): “We can decide to exit if the entropy is lower than a user-deﬁned threshold γ b : This strategy requires to ﬁne-tune an entropy threshold for each branch, by hand-tuning, heuristically [9], or us-ing some default choice [10].”
Deep Mixture of Experts via Shallow Embedding (2018, background): “Like the work on conditional computation [25], dynamic pruning relies on sample inefficient reinforcement learning techniques to train many convolutional gates.”
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking (2018, methodology): “Prior work has utilized this term to describe a property of the network as a whole, disregarding its internal state (Wang et al., 2017).”
TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learning (2019, background): “The TAFE-Net design is also related to works on dynamic neural networks [44, 48, 43, 27] which focus on dynamic execution at runtime.”
Dynamic-stride-net: deep convolutional neural network with dynamic stride (2019, background): “Moreover, network acceleration can be improved using cascaded models of various pretrained deep convolutional neural networks [15,16,17,18].”
Predict Responsibly: Increasing Fairness by Learning To Defer (2017, background): “Wang et al. (2017) propose a cascading model, which can be learned from end-to-end.”
Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer (2017, background): “[33] propose a cascading model, which can be learned end-to-end.”