Shallow-Deep Networks: Understanding and Mitigating Network Overthinking

By Yigitcan Kaya, Sanghyun Hong, Tudor Dumitras. 2018

395 citations recorded by the source corpus.

Subfields: Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Optimization & Training

Citations in the corpus

Consistent Accelerated Inference via Confident Adaptive Transformers (2021, background): “[20] Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras.”
Dynamic Neural Networks: A Survey (2021, methodology): “For example, multi-exit models can be used to: 1) alleviate the over-thinking issue while reducing the overall computation [50], [266]; 2) perform long-tailed classification [267] by inducing early exiting in the training stage; and 3) improve the model robustness [268].”
Why Should We Add Early Exits to Neural Networks? (2020, methodology): “For convolutional layers, this is slightly more complex because early layers in most CNNs have a very high dimensionality, which would result in extremely highdimensional classifiers [30].”
Pre-trained models for natural language processing: A survey (2020, methodology): “The idea of early exit is first applied in computer vision, such as BranchyNet [182] and Shallow-Deep Network [183].”
Don’t Shoot Butterfly with Rifles: Multi-Channel Continuous Speech Separation with Early Exit Transformer (2020, methodology): “2) The Transformer model may suffer from the “overthinking” problem [17] as it contains too many encoder layers.”
A Survey on Dynamic Neural Networks for Natural Language Processing (2022, background): “Early exit techniques aim to terminate model inference in early layers, to save computation and sometimes improve performance by resolving the overthinking problem (Kaya et al., 2019), i.”
Self-Distillation: Towards Efficient and Compact Neural Networks (2021, background): “Adaptive computation, which is based on the principle that neural networks should classify the images correctly at the least cost of computation and energy, is a rising star in the study of neural network acceleration [66], [67].”
UnNatural Language Inference (2020, background): “to the overthinking phenomenon commonly observed in deep neural networks (Kaya et al., 2019) and BERT-based models (Zhou et al.”
BERT Loses Patience: Fast and Robust Inference with Early Exit (2020, methodology): “ NLI tasks, and 2 on the Yelp sentiment analysis task. Also, PABEE increases the number of queries needed to attack by a large margin, providing more protection to the model. Compared to Shallow-Deep [5], our model demonstrates signiﬁcant robustness improvements. To analyze, although the early exit ”
SPINN: synergistic progressive inference of neural networks over device and cloud (2020, methodology): “We jointly train all classifiers from scratch and employ the cost function introduced in [35] as follows: L = N − 1 i = 0 τ i ∗ L i with τ i starting uniformly at 0 .”
PPFL: privacy-preserving federated learning with trusted execution environments (2021, background): “Consequently, these classifiers provide multiple early exits, one per layer, during the forward pass in inference [29].”
A Panda? No, It's a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network Inference (2020, methodology): “There are two types of input-adaptive DNNs: adaptive neural networks (AdNNs) and multi-exit architectures.”
Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks (2021, background): “Networks with variable numbers of test time iterations/layers have also been studied, including variable depth networks [Graves, 2016, Huang et al., 2016, Kaya et al., 2019, Eyzaguirre and Soto, 2020].”