DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
By Victor Sanh, Lysandre Debut, Julien Chaumond, Thomas Wolf. 2019
10,143 citations recorded by the source corpus.
As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive biases learned by larger models during pre-training, we introduce a triple loss combining language modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device study.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Yejin Choi
- Noah A. Smith
- Graham Neubig
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Ming-Hsuan Yang
- Andrew Zisserman
- Cordelia Schmid
In Efficiency & Compression
- Kurt Keutzer
- Dan Alistarh
- Zi-Mei Yao
- Alexander M. Rush
- A. Gholami
- Canwen Xu
- Jimmy J. Lin
- Michael W. Mahoney
Citations in the corpus
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “Some current work along these lines include distillation [Hinton et al., 2015; Sanh et al., 2019; Jiao et al., 2019], parameter sharing [Lan et al.”
- Pretrained Transformers Improve Out-of-Distribution Robustness (2020, methodology): “, 2020) and also a distilled version of BERT, DistilBERT (Sanh et al., 2019).”
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, methodology): “For example, we avoid using model distillation methods because they can add a significant computational overhead (Sanh et al., 2019; Turc et al., 2019) or cause a significant degradation in accuracy (Liu et al.”
- TinyBERT: Distilling BERT for Natural Language Understanding (2019, methodology): “DistilBERT (Sanh et al., 2019) performs distillation at pre-training stage on large-scale corpus.”
- Learning Compact Metrics for MT (2021, neutral): “Distillation has been shown to successfully transfer knowledge from large models to smaller ones (Hinton et al., 2015), but it requires access to a large corpus of un-labelled data (Sanh et al., 2019; Turc et al., 2019), which does not exist for our task.”
- Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning. (2021, background): “BERT’s strong performance has resulted in the Cambrian explosion of studies of the inner working mechanisms and various modifica-tions (Clark et al., 2019; de Vries et al., 2019; Lan et al., 2019; Liu et al., 2019; Sanh et al., 2019; Rad-ford et al.; Raffel et al., 2019; Yang et al., 2019).”
- HateCheck: Functional Tests for Hate Speech Detection Models (2020, methodology): “We demonstrate its utility by evaluating two DistilBERT models (Sanh et al., 2019), which Tran et al.”
- Self-Instruct: Aligning Language Models with Self-Generated Instructions (2022, background): “Knowledge distillation (Hinton et al., 2015; Sanh et al., 2019; West et al., 2021; Magister et al., 2022) often involves the transfer of knowledge from larger models to smaller ones.”
- Efficiently Scaling Transformer Inference (2022, neutral): “…attention layers (Roy et al., 2020; Choromanski et al., 2020; Kitaev et al., 2020; Sukhbaatar et al., 2019; Child et al., 2019), distillation (Sanh et al., 2019; Sun et al., 2020), and model compression techniques, such as pruning (Li et al., 2020b; Brix et al., 2020; Zhou et al., 2021; Li…”
- Re-contextualizing Fairness in NLP: The Case of India (2022, methodology): “We now demonstrate biases in the default HuggingFace sentiment pipeline which is DistilBERTbase-uncased (Sanh et al., 2019) fine-tuned on the SST-2 (Socher et al., 2013).11 We perform per- 9https://tinyurl.com/wiki-in-regions 10Broad (and overlapping) categories, not caste names.”
- Understanding Dataset Difficulty with V-Usable Information (2021, positive): “The four models we use are GPT2-small (Radford et al., 2019), BERT-base-cased (Devlin et al., 2019), DistilBERT-base-uncased (Sanh et al., 2019), and BART-base (Lewis et al., 2020).”
- Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, methodology): “These models were distilled with loss terms adapted from Sanh et al. (2019), Jiao et al. (2020), and Shleifer and Rush (2020), in order to match the predicted next-token distributions and latent states between these small models and BlenderBot3B during training.”
- Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “…more efficient models (Humeau et al., 2019) is a step in that direction, and so are works that explore smaller footprints for Transformer-based models, e.g. through knowledge distillation (Sanh et al., 2019), adaptive spans (Sukhbaatar et al., 2019), or layer pruning (Fan, Grave, and Joulin, 2019).”
- HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, positive): “The distillation of large models (e.g. DistilBERT (Sanh et al., 2019)) is one of the most promising directions.”