FastFormers: Highly Efficient Transformer Models for Natural Language Understanding

By Young Jin Kim, Hany Hassan Awadalla. 2020

49 citations recorded by the source corpus.

Transformer-based models are the state-of-the-art for Natural Language Understanding (NLU) applications. Models are getting bigger and better on various tasks. However, Transformer models remain computationally challenging since they are not efficient at inference-time compared to traditional approaches. In this paper, we present FastFormers, a set of recipes to achieve efficient inference-time performance for Transformer-based models on various NLU tasks. We show how carefully utilizing knowledge distillation, structured pruning and numerical optimization can lead to drastic improvements on inference efficiency. We provide effective recipes that can guide practitioners to choose the best settings for various NLU tasks and pretrained models. Applying the proposed recipes to the SuperGLUE benchmark, we achieve from 9.8x up to 233.9x speed-up compared to out-of-the-box models on CPU. On GPU, we also achieve up to 12.4x speed-up with the presented methods. We show that FastFormers can drastically reduce cost of serving 100 million requests from 4,223 USD to just 18 USD on an Azure F16s_v2 instance. This translates to a sustainable runtime by reducing energy consumption 6.9x - 125.8x according to the metrics used in the SustaiNLP 2020 shared task.

Subfields: Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

Sparse is Enough in Scaling Transformers (2021, methodology): “For example, [18] pretrains a large model and uses knowledge distillation along with pruning to get more than 10x faster inference.”
Visual Attention Methods in Deep Learning: An In-Depth Survey (2022, methodology): “Many methods propose efficient architectures for attention [91, 156, 176, 88].”
Block Pruning For Faster Transformers (2021, methodology): “Related to our approach, Kim and Awadalla (2020) and McCarley (2019) both apply structured pruning on the heads of the multi-head attention (MHA) and on the inner-layer nodes of the feed-forward network (FFN).”
LightSeq: A High Performance Inference Library for Transformers (2021, background): “However, the parameters of these models become increasingly large, which causes the high latency of inference and brings great challenges to the deployment (Kim and Hassan, 2020).”