Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning

By Bingbing Li, Zhenglun Kong, Tianyun Zhang, Ji Li, Z. Li, Hang Liu, Caiwen Ding. 2020

67 citations recorded by the source corpus.

Pretrained large-scale language models have increasingly demonstrated high accuracy on many natural language processing (NLP) tasks. However, the limited weight storage and computational speed on hardware platforms have impeded the popularity of pretrained models, especially in the era of edge computing. In this work, we propose an efficient transformer-based large-scale language representation using hardware-friendly block structure pruning. We incorporate the reweighted group Lasso into block-structured pruning for optimization. Besides the significantly reduced weight storage and computation, the proposed approach achieves high compression rates. Experimental results on different models (BERT, RoBERTa, and DistilBERT) on the General Language Understanding Evaluation (GLUE) benchmark tasks show that we achieve up to 5.0x with zero or minor accuracy degradation on certain task(s). Our proposed method is also orthogonal to existing compact pretrained language models such as DistilBERT using knowledge distillation, since a further 1.79x average compression rate can be achieved on top of DistilBERT with zero or minor accuracy degradation. It is suitable to deploy the final compressed model on resource-constrained edge devices.

Subfields: Artificial Intelligence, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Kurt Keutzer
2. A. Gholami
3. Sheng Shen
4. Sehoon Kim
5. Henryk Michalewski
6. Joseph Hassoun
7. Afroz Mohiuddin
8. David Thorsley

In Large Language Models

1. A. Gholami
2. Kurt Keutzer
3. Sehoon Kim
4. Sheng Shen
5. Aakanksha Chowdhery
6. Lukasz Kaiser
7. Henryk Michalewski

In Efficiency & Compression

1. Kurt Keutzer
2. A. Gholami
3. Sehoon Kim
4. Sheng Shen
5. Joseph Hassoun
6. David Thorsley
7. Aakanksha Chowdhery
8. Lukasz Kaiser

Citations in the corpus

• Efficiently Scaling Transformer Inference (2022, neutral): “…(Sanh et al., 2019; Sun et al., 2020), and model compression techniques, such as pruning (Li et al., 2020b; Brix et al., 2020; Zhou et al., 2021; Li et al., 2020a; Wang et al., 2020), or quantization (Dettmers et al., 2022; Abdolrashidi et al., 2021; Zafrir et al., 2019; Zhang et al., 2018).”
• Sparse is Enough in Scaling Transformers (2021, methodology): “Structured pruning based approaches [47, 22, 43] account for this challenge by leveraging sparsity in hardware in CPU and GPU architectures [1].”
• Learned Token Pruning for Transformers (2021, methodology): “2xlarge instance that has a NVIDIA V100 GPU with CUDA 11.1.”
• Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2020, unknown): “Unlike unstructured pruning, structured pruning focuses on pruning structured blocks of weights (Li et al., 2020a) or even complete architectural components in the BERT model, by reducing and simplifying certain numerical modules: • Attention head pruning.”