AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search

By Daoyuan Chen, Yaliang Li, Minghui Qiu, Zhen Wang, Bofang Li, Bolin Ding, Hongbo Deng, Jun Huang, Wei Lin, Jingren Zhou. 2020

112 citations recorded by the source corpus.

Large pre-trained language models such as BERT have shown their effectiveness in various natural language processing tasks. However, the huge parameter size makes them difficult to be deployed in real-time applications that require quick inference with limited resources. Existing methods compress BERT into small models while such compression is task-independent, i.e., the same compressed BERT for all different downstream tasks. Motivated by the necessity and benefits of task-oriented BERT compression, we propose a novel compression method, AdaBERT, that leverages differentiable Neural Architecture Search to automatically compress BERT into task-adaptive small models for specific tasks. We incorporate a task-oriented knowledge distillation loss to provide search hints and an efficiency-aware loss as search constraints, which enables a good trade-off between efficiency and effectiveness for task-adaptive BERT compression. We evaluate AdaBERT on several NLP tasks, and the results demonstrate that those task-adaptive compressed models are 12.7x to 29.3x faster than BERT in inference time and 11.5x to 17.0x smaller in terms of parameter size, while comparable performance is maintained.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Sheng Zhao
5. Kurt Keutzer
6. Michael W. Mahoney
7. A. Gholami
8. Enhong Chen

In Computer Vision and Pattern Recognition

1. Kurt Keutzer
2. Tao Qin
3. Tie-Yan Liu
4. Michael W. Mahoney
5. A. Gholami
6. Xu Tan
7. Jitendra Malik
8. Luc Van Gool

Citations in the corpus

• AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations (2020, methodology): “NAS has been widely used in different tasks, such as image classification[26], natural language understanding[4], etc. Joglekar[12] firstly utilized NAS into large scale recommendation models and proposed a novel type of embedding named as Multi-size Embedding (ME).”
• Lightspeech: Lightweight and Fast Text to Speech with Neural Architecture Search (2021, background): “Recently, neural architecture search (NAS) [13, 14] is leveraged to automatically design light weight models with promising performance [15, 16].”
• Big Little Transformer Decoder (2023, methodology): “Well-known approaches include efficient architecture design [25, 31, 34, 57, 64, 72], quantization [8, 29, 51, 71, 76, 78, 79], pruning [11, 13, 32, 33, 39, 46, 61], and neural architecture search [5, 53, 54, 63, 73, 77].”
• Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2020, unknown): “It is possible to either completely replace the Transformer backbone with a deep CNN network (Chen et al., 2020a), or to replace only a few encoder units to balance performance and efficiency (Tian et al.”
• Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search (2020, methodology): “Motivated by the promising performance achieved by those NAS works, recent works have extended NAS to more tasks such as segmentation [44, 10], image enhancement [20, 41], and language modeling [9].”