Deep Net Triage: Assessing the Criticality of Network Layers by Structural Compression

By Theodore S. Nowak, Jason J. Corso. 2018

3 citations recorded by the source corpus.

Deep network compression seeks to reduce the number of parameters in the network while maintaining a certain level of performance. Deep network distillation seeks to train a smaller network that matches soft-max performance of a larger network. While both regimes have led to impressive performance for their respective goals, neither provide insight into the importance of a given layer in the original model, which is useful if we are to improve our understanding of these highly parameterized models. In this paper, we present the concept of deep net triage, which individually assesses small blocks of convolution layers to understand their collective contribution to the overall performance, which we call \emph{criticality}. We call it triage because we assess this criticality by answering the question: what is the impact to the health of the overall network if we compress a block of layers into a single layer. We propose a suite of triage methods and compare them on problem spaces of varying complexity. We ultimately show that, across these problem spaces, deep net triage is able to indicate the of relative importance of different layers. Surprisingly, our local structural compression technique also leads to an improvement in overall accuracy when the final model is fine-tuned globally.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Jianping Gou
3. S. Maybank
4. Baosheng Yu
5. Xi Li
6. Hui Wang
7. Hanbin Zhao
8. Xu Tan

In Radiology, Nuclear Medicine and Imaging

1. Dacheng Tao
2. S. Maybank
3. Xi Li
4. Hui Wang
5. Baosheng Yu
6. Hanbin Zhao
7. Jianping Gou
8. Xu Tan

Citations in the corpus

• Knowledge Distillation: A Survey (2020, methodology): “Currently, most KD methods focus on new types of knowledge or distillation loss functions, leaving the design of theteacher–studentarchitecturespoorlyinvestigated(Nowak and Corso 2018; Crowley et al. 2018; Kang et al. 2020; Liu et al. 2019i; Ashok et al. 2018; Liu et al. 2019a).”
• Progressive Blockwise Knowledge Distillation for Neural Network Acceleration (2018, methodology): “Figure (c): Block Structure Comparison, the accuracy of our method is better than Deep Net’s [Nowak and Corso, 2018] at every stage.”