SqueezeBERT: What can computer vision teach NLP about efficient neural networks?

By Forrest N. Iandola, Albert Eaton Shaw, Ravi Krishna, Kurt Keutzer. 2020

143 citations recorded by the source corpus.

Humans read and write hundreds of billions of messages every day. Further, due to the availability of large datasets, large computing systems, and better neural network models, natural language processing (NLP) technology has made significant strides in understanding, proofreading, and organizing these messages. Thus, there is a significant opportunity to deploy NLP in myriad applications to help web users, social networks, and businesses. Toward this end, we consider smartphones and other mobile devices as crucial platforms for deploying NLP models at scale. However, today’s highly-accurate NLP neural network models such as BERT and RoBERTa are extremely computationally expensive, with BERT-base taking 1.7 seconds to classify a text snippet on a Pixel 3 smartphone. To begin to address this problem, we draw inspiration from the computer vision community, where work such as MobileNet has demonstrated that grouped convolutions (e.g. depthwise convolutions) can enable speedups without sacrificing accuracy. We demonstrate how to replace several operations in self-attention layers with grouped convolutions, and we use this technique in a novel network architecture called SqueezeBERT, which runs 4.3x faster than BERT-base on the Pixel 3 while achieving competitive accuracy on the GLUE test set. A PyTorch-based implementation of SqueezeBERT is available as part of the Hugging Face Transformers library: https://huggingface.co/squeezebert

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Sparse is Enough in Scaling Transformers (2021, background): “Several natural language models used for mobile inference [13, 39] rely on distillation [32] to speed up inference from the pretrained large models.”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, background): “Some of the complementary efforts in this area are compact and fast architectures by design (Sun et al., 2020; Iandola et al., 2020), weight sharing (Dehghani et al., 2018; Lan et al., 2019), parameter reuse across multiple downstream tasks (Houlsby et al., 2019; Stickland and Murray, 2019),…”
A Comparative Analysis of Pretrained Language Models for Text-to-Speech (2023, methodology): “Regarding transformer encoders for NLU, since the emergence of BERT, many BERT-like models have been released with more pretraining data [4, 5], novel pretraining methods [6], more efficient architectures [7], or smaller architectures with faster inference [8, 9, 10, 11].”
High Performance Natural Language Processing (2020, methodology): “Then, we dive into case studies by examining speciﬁc models such as Iandola et al. (2020) and Sun et al. (2020).”
ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning (2020, result): “The result of AdamW is directly from (Iandola et al. 2020) and the result of AdamW∗ is reproduced by us.”
Big Little Transformer Decoder (2023, methodology): “Well-known approaches include efficient architecture design [25, 31, 34, 57, 64, 72], quantization [8, 29, 51, 71, 76, 78, 79], pruning [11, 13, 32, 33, 39, 46, 61], and neural architecture search [5, 53, 54, 63, 73, 77].”
SqueezeLLM: Dense-and-Sparse Quantization (2023, unknown): “This can be broadly categorized into efficient architecture design [32, 38, 42, 60, 67, 73], pruning [18, 20, 40, 41, 48, 52, 65, 79], neural architecture search [7, 58, 59, 66, 74, 78], and quantization [35, 55, 81, 82].”
Pre-trained Summarization Distillation (2020, unknown): “, 2020) (Iandola et al., 2020) Discussion here”
MLPruning: A Multilevel Structured Pruning Framework for Transformer-based Models (2021, background): “These efforts can be generally categorized as follows: (i) knowledge distillation [21, 41, 35, 39]; (ii) quantization [3, 54, 37, 12, 53, 55, 2, 10]; (iii) new architecture design [40, 19, 23, 22, 45]; and (iv) pruning.”
Your fairness may vary: Pretrained language model fairness in toxic text classification (2021, background): “…terms of harms, a false negative (toxic text that is missed) may cause individuals to feel threatened or (Sun et al., 2020) 25.3M 95MB SqueezeBERT (Iandola et al., 2020) 51M ∗ 196MB DistilBERT (Sanh et al., 2020) 66M 256MB Regular BERT (Devlin et al., 2019) 110M 418MB ELECTRA (Clark et al., 2020)…”