HAT: Hardware-Aware Transformers for Efficient Natural Language Processing

By Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, Song Han. 2020

292 citations recorded by the source corpus.

Transformers are ubiquitous in Natural Language Processing (NLP) tasks, but they are difficult to be deployed on hardware due to the intensive computation. To enable low-latency inference on resource-constrained hardware platforms, we propose to design Hardware-Aware Transformers (HAT) with neural architecture search. We first construct a large design space with arbitrary encoder-decoder attention and heterogeneous layers. Then we train a SuperTransformer that covers all candidates in the design space, and efficiently produces many SubTransformers with weight sharing. Finally, we perform an evolutionary search with a hardware latency constraint to find a specialized SubTransformer dedicated to run fast on the target hardware. Extensive experiments on four machine translation tasks demonstrate that HAT can discover efficient models for different hardware (CPU, GPU, IoT device). When running WMT’14 translation task on Raspberry Pi-4, HAT can achieve 3× speedup, 3.7× smaller size over baseline Transformer; 2.7× speedup, 3.6× smaller size over Evolved Transformer with 12,041× less search cost and no performance loss. HAT is open-sourced at https://github.com/mit-han-lab/hardware-aware-transformers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Neural Architecture Search & AutoML, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Neural Architecture Search & AutoML

In ML Systems & Hardware

Citations in the corpus

Efficiently Scaling Transformer Inference (2022, neutral): “…(Sanh et al., 2019; Sun et al., 2020), and model compression techniques, such as pruning (Li et al., 2020b; Brix et al., 2020; Zhou et al., 2021; Li et al., 2020a; Wang et al., 2020), or quantization (Dettmers et al., 2022; Abdolrashidi et al., 2021; Zafrir et al., 2019; Zhang et al., 2018).”
Sparse is Enough in Scaling Transformers (2021, methodology): “Structured pruning based approaches [47, 22, 43] account for this challenge by leveraging sparsity in hardware in CPU and GPU architectures [1].”
Efficient Transformers: A Survey (2020, methodology): “Wang et al. (2020a) proposed HAT (Hardware-aware Transformers), a method that leverages NAS and uses hardware eﬃciency feedback as a reward signal.”
Transformers in Vision: A Survey (2021, methodology): “While Nerual Architecuter Search (NAS) has been well explored for CNNs to find an optimized architecture, it is relatively less explored in Transformers (even for language transformers [261], [262]).”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, background): “…(Houlsby et al., 2019; Stickland and Murray, 2019), knowledge distillation (Sanh et al., 2019; Jiao et al., 2020), neural architecture search (Guo et al., 2019; Wang et al., 2020a), pruning (Sanh et al., 2020; Prasanna et al., 2020), and better pre-training (Liu et al., 2019; Clark et al., 2020).”
DynaBERT: Dynamic BERT with Adaptive Width and Depth (2020, background): “Concurrently to our work, flexibility in both directions is also proposed in [27], but on the encoder-decoder Transformer structure [24] and on machine translation task.”
Representing Long-Range Context for Graph Neural Networks with Global Attention (2022, methodology): “Neural architecture search (NAS) was also applied to Transformer to fulﬁll the resource constraints for the edge devices [32].”
GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference (2020, methodology): “We evaluate GOBO on various datasets and state-of-theart attention-based NLP models: BERT (two variants), DistilBERT [2], RoBERTa (two variants) [10], HAT [11], and SpanBERT [12].”
Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution (2020, methodology): “To this end, researchers have proposed different techniques to reduce the computation cost (to around 10(2) GPU hours), such as differentiable architecture search [24], path-level binarization [4], single-path one-shot sampling [12, 6], and weight sharing [43, 2, 21, 50].”
Pay Attention when Required (2020, methodology): “Several optimization approaches, that used pruning (Michel et al., 2019), and distillation (Sanh et al., 2020; Jiao et al., 2020; Wang et al., 2020b), were able to achieve better run-time performance for an accuracy trade-off.”
LocalViT: Analyzing Locality in Vision Transformers (2021, unknown): “A parallel research strand incorporates global connectivity into the network via self-attention [4], [7], [8], [9], [10].”
EfficientFormer: Vision Transformers at MobileNet Speed (2022, methodology): “We directly optimize for inference speed instead of MACs or number of parameters [31, 32, 33].”
AutoFormer: Searching Transformers for Visual Recognition (2021, background): “For transformers, there are few studies applying NAS to improve their architectures [45, 55].”
SqueezeLLM: Dense-and-Sparse Quantization (2023, unknown): “This can be broadly categorized into efficient architecture design [32, 38, 42, 60, 67, 73], pruning [18, 20, 40, 41, 48, 52, 65, 79], neural architecture search [7, 58, 59, 66, 74, 78], and quantization [35, 55, 81, 82].”
Group Fisher Pruning for Practical Network Compression (2021, methodology): “There exist methods (Wang et al., 2020; Li et al., 2020) that directly profile the running time without resorting to proxies.”
SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning (2020, methodology): “Besides the experiments above that leverage existing model architecture, we also explore the potentials of co-designing SpAtten with model architecture by searching a Hardware-Aware Transformer (HAT) [20] for SpAtten-e2e.”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, methodology): “Hardware-aware transformer designs using NAS techniques for edge-compatible and low-latency inference are presented in [68].”