ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGA

By Song Han, Junlong Kang, Huizi Mao, Yiming Hu, Xin Li, Yubin Li, Dongliang Xie, Hong Luo, Song Yao, Yu Wang and 2 more. 2016

659 citations recorded by the source corpus.

Long Short-Term Memory (LSTM) is widely used in speech recognition. In order to achieve higher prediction accuracy, machine learning scientists have built increasingly larger models. Such large model is both computation intensive and memory intensive. Deploying such bulky model results in high power consumption and leads to a high total cost of ownership (TCO) of a data center. To speedup the prediction and make it energy efficient, we first propose a load-balance-aware pruning method that can compress the LSTM model size by 20x (10x from pruning and 2x from quantization) with negligible loss of the prediction accuracy. The pruned model is friendly for parallel processing. Next, we propose a scheduler that encodes and partitions the compressed model to multiple PEs for parallelism and schedule the complicated LSTM data flow. Finally, we design the hardware architecture, named Efficient Speech Recognition Engine (ESE) that works directly on the sparse LSTM model. Implemented on Xilinx KU060 FPGA running at 200MHz, ESE has a performance of 282 GOPS working directly on the sparse LSTM network, corresponding to 2.52 TOPS on the dense one, and processes a full LSTM for speech recognition with a power dissipation of 41 Watts. Evaluated on the LSTM for speech recognition benchmark, ESE is 43x and 3x faster than Core i7 5930k CPU and Pascal Titan X GPU implementations. It achieves 40x and 11.5x higher energy efficiency compared with the CPU and GPU respectively.

Subfields: Artificial Intelligence, Hardware and Architecture

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Hardware and Architecture

Citations in the corpus

AMC: AutoML for Model Compression and Acceleration on Mobile Devices (2018, methodology): “Fine-grained pruning method prunes neural networks based on individual connections to achieve sparsity in both weights and activations, which is able to achieve higher compression ratio and can be accelerated with specialized hardware such as [18, 17, 39].”
Deep Rewiring: Training very sparse deep networks (2017, background): “Memory is also the bottleneck in terms of energy consumption in TPUs and FPGAs (Han et al., 2017; Iandola et al., 2016).”
Exploring the Regularity of Sparse Structure in Convolutional Neural Networks (2017, background): “Fine-grained sparsity has been proven to work well on a wide range of popular CNN and RNN neural network models [7, 8, 15, 16].”
Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2019, methodology): “Therefore, hardware architectures should be designed to work directly for optimized models [145].”
A^3: Accelerating Attention Mechanisms in Neural Networks with Approximation (2020, background): “CNN accelerators for custom ASIC or FPGA device [1], [2], [3], [61], [62], [63], [64], [65], [66], [67] and RNN accelerators [3], [68], [69], [70] are proposed to accelerate NN processing by exploiting massive parallelism and optimizing data movements.”
Balanced Sparsity for Efficient DNN Inference on GPU (2018, background): “( Wen et al. 2016; Wang et al. 2018) Therefore, customized hard-wares (Han et al. 2016; 2017; Parashar et al. 2017) are required for the widely deployment of model pruning.”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, methodology): “The minimal gated unit (MGU) integrates the reset gate and update gate of the GRU [87].”
Hardware-Guided Symbiotic Training for Compact, Accurate, yet Execution-Efficient LSTM (2019, methodology): “Follow-up works have successfully scaled this technique to LSTMs [5], [16], [17].”
PruneTrain: Gradual Structured Pruning from Scratch for Faster Neural Network Training (2019, background): “Such accesses and extra index operations lead to poor performance on deep learning accelerators with vector or matrix computing units despite reducing the number of parameters and FLOPs (Han et al., 2017; Yu et al., 2017; Anwar et al., 2017).”
A survey of FPGA-based accelerators for convolutional neural networks (2018, methodology): “A comparison of weight matrix and makespans in (a) naive pruning and (b) load-balance aware pruning [66] After this, they quantize 32b FP weights to 12b integer.”
FPGA/DNN Co-Design: An Efficient Design Methodology for 1oT Intelligence on the Edge (2019, methodology): “t they only consider the DNN inference latency on CPUs and GPUs, not on FPGAs. On the other hand, for FPGA-based DNN accelerator, recent technologies such as quantization [2, 4] and model compression [15] are used to reduce DNN model size, and latency-directed resource allocation [3] and fine-graine”
A graph-based CNN-LSTM stock price prediction algorithm with leading indicators (2021, methodology): “It has been successfully used in various fields such as machine translation [50], speech recognition [51], image description generation [52], video tagging [53], and financial time series [54].”
SeerNet: Predicting Convolutional Neural Network Feature-Map Sparsity Through Low-Bit Quantization (2019, background): “Specialized accelerators on FPGA or ASIC show great potential in bringing a much higher speedup and power efﬁciency for sparse DNNs [6, 2, 21, 1, 28].”
MAERI: Enabling Flexible Dataflow Mapping over DNN Accelerators via Reconfigurable Interconnects (2018, background): “This work also opens up exciting opportunities in compiler designs that can take arbitrary DNNs and map them efficiently over a Maeri-like fabric.”
FPGA-Based Accelerators of Deep Learning Networks for Learning and Classification: A Review (2019, background): “model with dramatically reducing data bit-width [99], it is expected that future CNN models will get more complex with larger number of layers as the amount of training data”
AccPar: Tensor Partitioning for Heterogeneous Deep Learning Accelerators (2020, neutral): “…[36], OpenCL-FPGA-CNN [37] [126], Caffeine [38], DeepBurning [39], FPGA-DPCNN [40], TABLA [41], DNNWEAVER [42], FP-DNN [43], FPGA-LSTM [44], ESE [45], FPGA-Dataﬂow [46], FPGA-BNN [47], FPGA-Utilization [48], FFT-CNN [49], VIBNN [50], iSwitch [127], Shortcut-Mining [128], FA3C [129], E-RNN [130]”