EIE: Efficient Inference Engine on Compressed Deep Neural Network
By Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A. Horowitz, William (Bill) J. Dally. 2016
2,611 citations recorded by the source corpus.
State-of-the-art deep neural networks (DNNs) have hundreds of millions of connections and are both computationally and memory intensive, making them difficult to deploy on embedded systems with limited hardware resources and power budgets. While custom hardware helps the computation, fetching weights from DRAM is two orders of magnitude more expensive than ALU operations, and dominates the required power. Previously proposed 'Deep Compression' makes it possible to fit large DNNs (AlexNet and VGGNet) fully in on-chip SRAM. This compression is achieved by pruning the redundant connections and having multiple connections share the same weight. We propose an energy efficient inference engine (EIE) that performs inference on this compressed network model and accelerates the resulting sparse matrix-vector multiplication with weight sharing. Going from DRAM to SRAM gives EIE 120x energy saving, Exploiting sparsity saves 10x, Weight sharing gives 8x, Skipping zero activations from ReLU saves another 3x. Evaluated on nine DNN benchmarks, EIE is 189x and 13x faster when compared to CPU and GPU implementations of the same DNN without compression. EIE has a processing power of 102 GOPS working directly on a compressed network, corresponding to 3 TOPS on an uncompressed network, and processes FC layers of AlexNet at 1.88x104 frames/sec with a power dissipation of only 600mW. It is 24,000x and 3,400x more energy efficient than a CPU and GPU respectively. Compared with DaDianNao, EIE has 2.9x, 19x and 3x better throughput, energy efficiency and area efficiency.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Yi Yang
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Alan L. Yuille
In Electrical and Electronic Engineering
- Song Han
- Carole-Jean Wu
- Young Geun Kim
- Joel S. Emer
- Tien-Ju Yang
- Tushar Krishna
- Vivienne Sze
- Sparsh Mittal
In Efficiency & Compression
Citations in the corpus
- Chasing Carbon: The Elusive Environmental Footprint of Computing (2020, background): “Their work includes system-performance increases, power management and energy-efficiency optimization through dynamic voltage and frequency scaling (DVFS) [96], [97], [101]–[111], and specialized hardware [4]–[9].”
- Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding (2015, methodology): “Based on “deep compression”, the EIE hardware accelerator Han et al. (2016) was later proposed that works on the compressed model, achieving significant speedup and energy efficiency improvement.”
- SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size (2016, methodology): “When we use Fire modules we set s 1 x 1 to be less than ( e 1 x 1 + e 3 x 3 ), so the squeeze layer helps to limit the number of input channels to the 3x3 filters, as per Strategy 2 from Section 3.1.”
- Pruning Filters for Efficient ConvNets (2016, methodology): “Modern libraries that provide speedup using sparse operations over CNNs are often limited (Szegedy et al. (2015a); Liu et al. (2015)) and maintaining sparse data structures also creates an additional storage overhead which can be significant for low-precision weights.”
- To prune, or not to prune: exploring the efficacy of pruning for model compression (2017, methodology): “The resulting pruned model typically has sparse connection matrices, so efficient inference using these sparse models requires purpose-built hardware capable of loading sparse matrices and/or performing sparse matrix-vector operations (Zhang et al., 2016; Han et al., 2016; Parashar et al., 2017).”
- Similarity-Preserving Knowledge Distillation (2019, background): “Unstructured sparsity usually results in better accuracy but requires specialized sparse matrix multiplication libraries [26] or hardware engines [10] in practice.”
- Deep Learning for IoT Big Data and Streaming Analytics: A Survey (2017, methodology): “In [185], an inference engine, called EIE, was designed with a special hardware architecture and SRAMs instead”
- Network Pruning via Transformable Architecture Search (2019, methodology): “Since unstructured pruning methods aim to make a big network sparse instead of changing the whole structure of the network, they need dedicated design for dependencies [8] and specific hardware to speedup the inference procedure.”
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices (2018, methodology): “Fine-grained pruning method prunes neural networks based on individual connections to achieve sparsity in both weights and activations, which is able to achieve higher compression ratio and can be accelerated with specialized hardware such as [18, 17, 39].”
- Learning to Prune Filters in Convolutional Neural Networks (2018, methodology): “In order to get real compression and speedup, it requires specially designed software (like sparse BLAS library) or hardware [10] to handle the sparsity.”
- Channel Pruning for Accelerating Very Deep Neural Networks (2017, neutral): “Sparse connection eliminates connections between neurons [17, 33, 29, 15, 14].”
- Dynamic Network Surgery for Efficient DNNs (2016, background): “It has also been validated that the sparse matrix-vector multiplication can further be accelerated by certain hardware design, making it more efficient than traditional CPU and GPU calculations [7].”
- Rigging the Lottery: Making All Tickets Winners (2019, background): “Han et al. (2016b) went further and introduced multiple rounds of magnitude pruning and retraining.”
- Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning (2020, background): “Sparse matrix multiplication is difficult to optimize on current GPU architectures (Han et al., 2016), although progress is being made.”
- Superneurons: dynamic GPU memory management for training deep neural networks (2018, background): “mory saving opportunities and computation efficiency for ignoring the memory variations among layers. Removing the parameter redundancy also reduces the memory usage. For example, the network pruning [11, 12] 11 removes near zero parameters; and quantization [24] or precision reduction [16] utilize ”
- Movement Pruning: Adaptive Sparsity by Fine-Tuning (2020, methodology): “Han et al. [2015] use”
- SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping (2020, background): “Existing work have sought to reduce memory consumption by using lower-precision floating points [12, 24], or compressing model parameters via quantization and spar-sification [3, 9, 13–15 , 20].”