Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks

By Minsoo Rhu, Mike O'Connor, Niladrish Chatterjee, Jeff Pool, S. Keckler. 2017

193 citations recorded by the source corpus.

Popular deep learning frameworks require users to fine-tune their memory usage so that the training data of a deep neural network (DNN) fits within the GPU physical memory. Prior work tries to address this restriction by virtualizing the memory usage of DNNs, enabling both CPU and GPU memory to be utilized for memory allocations. Despite its merits, virtualizing memory can incur significant performance overheads when the time needed to copy data back and forth from CPU memory is higher than the latency to perform DNN computations. We introduce a high-performance virtualization strategy based on a "compressing DMA engine" (cDMA) that drastically reduces the size of the data structures that are targeted for CPU-side allocations. The cDMA engine offers an average 2.6x (maximum 13.8x) compression ratio by exploiting the sparsity inherent in offloaded data, improving the performance of virtualized DNNs by an average 53% (maximum 79%) when evaluated on an NVIDIA Titan Xp.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Hardware and Architecture

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Hardware and Architecture

Citations in the corpus

NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks (2022, background): “Activations are often used to determine which units are more important for a given task in problems such as CL (Jung et al., 2020a; Golkar et al., 2019), network pruning (Kurtz et al., 2020; Hu et al., 2016; Rhu et al., 2018), and model interpretation (Erhan et al., 2009; Zeiler & Fergus, 2014).”
MASR: A Modular Accelerator for Sparse RNNs (2019, methodology): “We then simplify the neural networks to establish an efﬁcient baseline RNN for ASR.”
AntMan: Dynamic Scaling on GPU Clusters for Deep Learning (2020, methodology): “To optimize the limited and valued GPU memory for supporting larger batch-size DNN training, vDNN [39], Capuchin [36], CDMA [40], and Gist [26] adopts eviction, prefetching, and re-computation to reduce the GPU memory footprint, leveraging applicationspecific knowledge.”
Analyzing Machine Learning Workloads Using a Detailed GPU Simulator (2018, background): “2) Simulation: Some prior work has also simulated machine learning workloads, but these papers used private simulators [33]–[36].”
Low-Memory Neural Network Training: A Technical Report (2019, background): “Similarly, the activations can be sparsiﬁed, as in (Liu et al., 2019, Rhu et al., 2018).”
Gist: Efficient Data Encoding for Deep Neural Network Training (2018, methodology): “CDMA, designed on top of vDNN, leverages sparsity to compress the data sent between CPU and GPU [40].”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, background): “Although the use of on-GPU compression such as [201], [233], [250], and [251] can improve the performance, there is still a big gap far from our expectation due to the redundant design of general-purpose processors for ﬂexible programmability and general applicability.”
Capuchin: Tensor-based GPU Memory Management for Deep Learning (2020, methodology): “CDMA [25] and Gist [13] compress the data by leveraging the sparsity in particular neural network architecture, such as ReLU-Pool, which are all lossless compressing methods.”
SIGMA: A Sparse and Irregular GEMM Accelerator with Flexible Interconnects for DNN Training (2020, methodology): “We use a Bitmap format within SIGMA, where each element has a corresponding bit to indicate if a given element is zero or non-zero in the corresponding matrix [17], [24], [35], [36].”
RecNMP: Accelerating Personalized Recommendation with Near-Memory Processing (2019, unknown): “patterns found in fully-connected (FC), convolution (CNN), and recurrent (RNN) neural networks [3], [6]–[8], [11], [12], [15], [17], [18], [24], [33], [36], [40], [47], [53]–[55], [57], [60], [68], [71]–[74], [76], [79], [82].”
AccPar: Tensor Partitioning for Heterogeneous Deep Learning Accelerators (2020, methodology): “…[76], PIM-DNN [77], Scalpel [78], CirCNN [79], CoSMIC [80], SnaPEA [81], OLAcce [82], Prediction-based DNN [83], PERMDNN [84], MnnFast [141], DNN Computation Reuse [142], vDNN [143], Compressing-DMA-Engine [144], AxTrain [145], Eager pruning [146], Bit-Tactical [147], GENESYS [148] Emerging Tech.”
Inducing and Exploiting Activation Sparsity for Fast Neural Network Inference (2020, methodology): “This observation has been leveraged by hardware accelerators, e.g. (Albericio et al., 2016; Han et al., 2016; Parashar et al., 2017), and reference (Rhu et al., 2018) performed an analysis of naturally-occurring activation sparsity.”
Buddy Compression: Enabling Larger Memory for Deep Learning and HPC Workloads on GPUs (2019, background): “Many domain-speciﬁc solutions across the stack have been proposed to deal with this memory capacity challenge in deep learning training [36, 37, 38, 39, 18, 5, 40, 8].”
Machine Learning at Facebook: Understanding Inference at the Edge (2019, background): “Recent work on hardware for machine learning and efficient training and inference has substantially advanced the state-of-the-art [41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66].”