LIT: Learned Intermediate Representation Training for Model Compression

By Animesh Koratana, Daniel Kang, Peter Bailis, Matei A. Zaharia. 2019

69 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Ensemble Distillation for Robust Model Fusion in Federated Learning (2020, background): “By encouraging the student model to approximate the output logits of the teacher model, the student is able to imitate the teacher’s behavior with marginal quality loss (Romero et al., 2015; Zagoruyko & Komodakis, 2016; Kim et al., 2018; Tung & Mori, 2019; Koratana et al., 2019; Huang & Wang, 2017; ”
Contrastive Representation Distillation (2019, background): “Other papers (Yim et al., 2017; Huang & Wang, 2017; Kim et al., 2018; Yim et al., 2017; Huang & Wang, 2017; Ahn et al., 2019; Koratana et al., 2019) have enforced various criteria based on representations.”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, background): “Several other papers have enforced criteria based on multi-layer representations of a network [53, 20, 24, 3, 28].”
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (2020, background): “This loss function needs to be combined with task-speciﬁc loss [33, 17].”
Compressing Visual-linguistic Model via Knowledge Distillation (2021, background): “For example, [25, 6] transfer the learned knowledge by mimicking the mass function of the output probability across classes, or by minimizing the divergence of intermediate features [78, 36, 28, 77, 73].”
SEED: Self-supervised Distillation For Visual Representation (2021, background): “Similarly, works in (Ahn et al., 2019; Yim et al., 2017; Koratana et al., 2019; Huang & Wang, 2017) have utilized different learning objectives including consistency on feature maps, consistency on probability mass function, and maximizing the mutual information.”
Multi-label Contrastive Predictive Coding (2020, background): “Two baselines are considered: the original KD objective in [20] and the state-of-the-art CRD objective in [47], since other baselines [29, 1, 23, 26] are shown to have inferior performance in general.”
Online Ensemble Model Compression Using Knowledge Distillation (2020, methodology): “nvolves minimizing a combination of L2 loss between the features maps at an immediate layer of the two models and the regular KD (KL divergence loss) between the output distributions. Koratana et al. [22] extend this concept by comparing feature maps at not just one but multiple intermediate locatio”
Meta Knowledge Distillation (2022, background): “Other papers (Park et al., 2019; Yim et al., 2017; Huang & Wang, 2017; Kim et al., 2018; Ahn et al., 2019; Koratana et al., 2019) have proposed various distillation criteria based on representations.”
Training convolutional neural networks with cheap convolutions and online distillation (2019, background): “The extra supervision extracted from a pre-trained teacher model is often in form of class posterior probabilities [30], feature representations [31, 32, 40, 4], distribution of intermediate activations [41, 42], or inter-layer flow [43]).”
Few-Shot Learning With a Strong Teacher (2021, background): “The dark knowledge for KD can be in the form of soft labels [30], hidden layer activation [87], [88], and comparison relationships [89].”
Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation (2021, methodology): “The intermediate layer distillation methods were then introduced, so such methods utilize the teacher network’s knowledge from either the convolutional layer with feature-map level preserving localities [26, 36, 14, 34, 16]; or penultimate layer [24, 29, 21, 23, 30].”
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning (2022, background): “Network distillation [22, 30] constructs a student network and force it to have same output distribution of the teacher network over a chosen dataset.”