Persistent RNNs: Stashing Recurrent Weights On-Chip

By G. Diamos, Shubho Sengupta, Bryan Catanzaro, Mike Chrzanowski, Adam Coates, Erich Elsen, Jesse Engel, Awni Y. Hannun, Sanjeev Satheesh. 2016

95 citations recorded by the source corpus.

Subfields: Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Efficient Neural Audio Synthesis (2018, methodology): “We sidestep the overhead by implementing custom GPU operations (Diamos et al., 2016) for the sampling process.”
DiffWave: A Versatile Diffusion Model for Audio Synthesis (2020, background): “1 × faster than real-time on an Nvidia V100 GPU without engineering optimization.”
Deep Voice: Real-time Neural Text-to-Speech (2017, methodology): “To get close to real-time on a GPU, we instead build a kernel using the techniques of persistent RNNs (Diamos et al., 2016) which generates all samples in the output audio in a single kernel launch.”
Exploring Sparsity in Recurrent Neural Networks (2017, neutral): “Persistent Recurrent Neural Networks (Diamos et al., 2016) is a technique that increases the computational intensity of evaluating an RNN by caching the weights in on-chip memory such as caches, block RAM, or register ﬁles across multiple timesteps.”
Demystifying Parallel and Distributed Deep Learning (2018, background): “As opposed to feed-forward networks, RNNs contain self-connections and thus cannot be trained with backpropagation alone.”
Triton: an intermediate language and compiler for tiled neural network computations (2019, background): “Unfortunately, these libraries only support a restricted set of tensor operations, leaving the implementation of novel primitives to experts [13, 17, 25].”
Training RNNs as Fast as CNNs (2017, background): “Various strategies have been proposed to speed up recurrent networks (Diamos et al., 2016; Kuchaiev and Ginsburg, 2017; Shazeer et al., 2017).”
A Lightweight Recurrent Network for Sequence Modeling (2019, background): “Diamos et al. (2016) emphasize persistent computational kernels to exploit GPU’s inverted memory hierarchy for reusing/caching purpose.”
Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks (2017, background): “Nonetheless, we believe our proposal is equally applicable for some popular recurrent neural networks that extensively employ sparsity-inducing ReLU layers, including the GEMV-based (general-matrix-vector-multiplication) RNNs employed by Baidu for speech recognition [41] and language translation [42”
Timeloop: A Systematic Approach to DNN Accelerator Evaluation (2019, background): “Deep neural networks (DNNs) have emerged as a key approach for solving complex problems across a wide range of domains, including image recognition [21], speech processing [3], [14], [16], natural language processing [9], language translation [11], and autonomous navigation [23].”
A Configurable Cloud-Scale DNN Processor for Real-Time AI (2018, methodology): “Baidu’s Persistent RNN [62] uses a similar approach targeting GPU devices for distributed training, although GPUs’ on-chip memory is more limited in capacity and inﬂexible in precision.”
Virtualizing Deep Neural Networks for Memory-Efficient Neural Network Design (2016, methodology): “hese frameworks is that the DRAM capacity limits of the GPU(s) in the system eventually limit the size the of the DNN that can be trained (Section II-C). To work around the memory capacity bottleneck [10, 11], ML practitioners must either use less desirable DNN architectures (e.g., smaller number of”
Recurrent Memory Array Structures (2016, background): “Performance analysis supports persistent RNN paper (Diamos et al., 2016); Most time is spent moving data, pointwise OPs are very cheap”
SmoothOut: Smoothing Out Sharp Minima to Improve Generalization in Deep Learning (2018, background): “Recently, one observation is that large-batch SGD has worse generalization than small-batch SGD [1][2][3].”
SCNN: An accelerator for compressed-sparse convolutional neural networks (2017, background): “Driven by the availability of massive data and the computational capability to process it, deep learning has recently emerged as a critical tool for solving complex problems across a wide range of domains, including image recognition [22], speech processing [3, 14, 18], natural language processing [”
Low latency RNN inference with cellular batching (2018, unknown): “Persistent RNN [19] exploits the weight-sharing feature of RNN to accelerate the computation.”
Compression of Deep Learning Models for Text: A Survey (2020, methodology): “[15] showed that for small models, one can cache the RNN weights in on-chip memory such as caches, block RAM, or register files across multiple timesteps.”
SmoothOut: Smoothing Out Sharp Minima for Generalization in Large-Batch Deep Learning (2018, background): “However, accuracy loss has been widely observed in large-batch deep learning [2][7][8].”
Simple Recurrent Units for Highly Parallelizable Recurrence (2017, methodology): “, 2017) and to speed up recurrent networks (Diamos et al., 2016; Shazeer et al., 2017; Kuchaiev and Ginsburg, 2017).”