Moonshine: Distilling with Cheap Convolutions

By Elliot J. Crowley, Gavia Gray, Amos J. Storkey. 2017

127 citations recorded by the source corpus.

Many engineers wish to deploy modern neural networks in memory-limited settings; but the development of flexible methods for reducing memory use is in its infancy, and there is little knowledge of the resulting cost-benefit. We propose structural model distillation for memory reduction using a strategy that produces a student architecture that is a simple transformation of the teacher architecture: no redesign is needed, and the same hyperparameters can be used. Using attention transfer, we provide Pareto curves/tables for distillation of residual networks with four benchmark datasets, indicating the memory versus accuracy payoff. We show that substantial memory savings are possible with very little loss of accuracy, and confirm that distillation provides student network performance that is better than training that student architecture directly on data.

Subfields: Industrial and Manufacturing Engineering, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Industrial and Manufacturing Engineering

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Similarity-Preserving Knowledge Distillation (2019, methodology): “[4] proposed to replace regular convolutions in the teacher network with cheaper grouped and pointwise convolutions in the student.”
Zero-shot Knowledge Transfer via Adversarial Belief Matching (2019, background): “Since then, the focus has mostly been on improving distillation efﬁciency by designing better students (Romero et al., 2015; Crowley et al., 2018), or getting small performance gains with extra loss terms, such as attention transfer (AT) (Zagoruyko and Komodakis, 2016a).”
Knowledge Distillation: A Survey (2020, methodology): “Currently, most KD methods focus on new types of knowledge or distillation loss functions, leaving the design of theteacher–studentarchitecturespoorlyinvestigated(Nowak and Corso 2018; Crowley et al. 2018; Kang et al. 2020; Liu et al. 2019i; Ashok et al. 2018; Liu et al. 2019a).”
Relational Knowledge Distillation (2019, methodology): “In general, conventional KD methods [1, 2, 8, 11, 12, 25, 27, 45, 47] can commonly be expressed as minimizing the objective function:”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “The choices of teacher models are also well studied, where [227, 265] use multiple teacher models to jointly supervise the student model, while [43, 269] apply self-distillation without an extra teacher model.”
Training convolutional neural networks with cheap convolutions and online distillation (2019, unknown): “In [73], WRN-40-2 is used to be a teacher model, which is significantly larger than WRN-40-1.”
Search for Better Students to Learn Distilled Knowledge (2020, background): “This work [4] shows that such student is not optimal to learn the dark knowledge.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “To tackle these challenges, several novel self-distillation frameworks [43], [44], [57], [72], [93], [117], [145], [164], [238], [246], [265] have been proposed recently.”
Emotion Recognition in Speech using Cross-Modal Transfer in the Wild (2018, background): “logits [7], or by minimising cross entropy between both models’ probabilistic outputs [43], often through a high-temperature softmax that softens the predictions of each model [19, 34].”
Membership Inference Attacks on Machine Learning: A Survey (2021, methodology): “It allows the smaller student model to have similar accuracy to their teacher models [32].”
Separable Layers Enable Structured Efficient Linear Substitutions (2019, unknown): “, 2017), Moonshine (Crowley et al., 2018), Wide ResNet (Zagoruyko & Komodakis, 2016), ResNeXt (Xie et al.”
Large-Scale Generative Data-Free Distillation (2020, methodology): “The knowledge distillation approach has been an instrumental technique for model compression [6, 19], as well as for pushing state of the art by harnessing large amounts of unlabeled data to improve the performance of student models such as Noisy Student [54] and many others, see [52] for a survey.”
Interactive Knowledge Distillation (2020, unknown): “We replace the “Conv3×3-BN-ReLU” with “group Conv3×3-BN-ReLU-Conv1×1-BN-ReLU” to achieve the cheap ResNet, as suggested by [4].”
Knowledge Distillation from Few Samples (2018, background): “For instance, Crowley et al. (2017) uses cheap group convolutions and pointwise convolutions to build a small student-net and adopts knowledge distillation to transfer knowledge from a full-sized "teacher-net" to the "student-net".”