Model compression

By Cristian Bucila, R. Caruana, Alexandru Niculescu-Mizil. 2006

2,875 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

Citations in the corpus

FitNets: Hints for Thin Deep Nets (2014, background): “In Bucila et al. (2006), authors propose to train a neural network to mimic the output of a complex and large ensemble.”
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (2019, background): “Knowledge distillation [Bucila et al., 2006, Hinton et al., 2015] is a compression technique in which a compact model - the student - is trained to reproduce the behaviour of a larger model - the teacher - or an ensemble of models.”
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding (2019, background): “…a process of distilling or transferring the knowledge from a (set of) large, cumbersome model(s) to a lighter, easier-to-deploy single model, without signiﬁcant loss in performance (Bucilu et al., 2006; Hinton et al., 2015; Balan et al., 2015; Ba et al., 2016; Chen et al., 2015; Tan et al., 2019).”
Distilling the Knowledge in a Neural Network (2015, methodology): “We show later that matching the logits of the cumbersome model is actually a special case of distillation.”
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, methodology): “Motivated by these results, we propose a way of applying knowledge distillation (Bucilu et al., 2006; Ba and Caruana, 2014; Hinton et al., 2015) so that single-task models effectively teach a multi-task model.”
Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, background): “The idea of knowledge distillation (KD) dates back to [4].”
Adversarial Examples Are Not Bugs, They Are Features (2019, background): “There is a line of work, known as model distillation [HVD14; Fur+18; BCN06], where the goal is to train a new model to mimic another already trained model.”
Emerging Properties in Self-Supervised Vision Transformers (2021, methodology): “When using soft labels, the approach is often referred to as knowledge distillation [7, 35] and has been primarily designed to train a small network to mimic the output of a larger network to compress models.”
Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “In another class of methods called knowledge distillation, a small network is trained to mimic a much larger network [Bucila et al., 2006, Hinton et al., 2015].”
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (2021, background): “needed in inference (Buciluǎ et al., 2006; Hinton et al., 2015; Urban et al., 2017; Karnin, 1990; Reed, 1993; Augasta and Kathirvalavakumar, 2013; Liu et al., 2014; Han et al., 2015; Molchanov et al., 2017). Gordon et al. (2020) have demonstrated that overparmeterization can be exploited in finetuni”
Born Again Neural Networks (2018, background): “In the neural network community, similar ideas emerged in papers on model compression by Bucilua et al. (2006) and related work on knowledge distillation (KD) by Hinton et al. (2015).”
Towards A Rigorous Science of Interpretable Machine Learning (2017, background): “…quantiﬁable proxy: a researcher might ﬁrst claim that some model class—e.g. sparse linear models, rule lists, gradient boosted trees—are interpretable and then present algorithms to optimize within that class (e.g. Bucilu et al. [2006], Wang et al. [2017], Wang and Rudin [2015], Lou et al. [2012]).”
Do Deep Nets Really Need to be Deep? (2013, background): “The main idea behind model compression is to train a compact model to approximate the function learned by a larger, more complex model.”
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (2016, background): “perts [28], can further improve the performance. The ensemble has Mtimes more parameters than a single network; for memory-constrained applications, the ensemble can be distilled into a simpler model [10, 26]. It would be also interesting to investigate so-called implicit ensembles the where ensembl”
Compressing Neural Networks with the Hashing Trick (2015, methodology): “ and memory overhead, but evaluation speed during test time. HashedNets is complementary to this research, and the two approaches could be used in combination. Following the line of model compression,Bucilu et al. (2006),Hinton et al.(2014) andBa & Caruana(2014) recently introduce approaches to ”