Optimizing Performance of Recurrent Neural Networks on GPUs

By Jeremy Appleyard, Tomás Kociský, Phil Blunsom. 2016

95 citations recorded by the source corpus.

As recurrent neural networks become larger and deeper, training times for single networks are rising into weeks or even months. As such there is a significant incentive to improve the performance and scalability of these networks. While GPUs have become the hardware of choice for training and deploying recurrent models, the implementations employed often make use of only basic optimizations for these architectures. In this article we demonstrate that by exposing parallelism between operations within the network, an order of magnitude speedup across a range of network sizes can be achieved over a naive implementation. We describe three stages of optimization that have been incorporated into the fifth release of NVIDIA's cuDNN: firstly optimizing a single cell, secondly a single layer, and thirdly the entire network.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In ML Systems & Hardware

Citations in the corpus

IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, methodology): “For LSTM-based agents, we also obtain signiﬁcant speedups on the learner by exploiting the network structure and op-LSTM eration fusion for LSTMs (Appleyard et al., 2016) (see Figure 3c).”
TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models (2021, methodology): “In deep learning, it has been used to accelerate the computation of multi-layer RNNs on a single GPU (Appleyard et al., 2016), where different input positions of different layers can execute in parallel in a wavefront fashion to maximize the utilization of the GPU.”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “since RNN units are usually chained together (forming consecutive recurrent layers), two types of concurrency can be considered: within the same layer, and between consecutive layers. Appleyard et al.[7] describe several optimizations that can be implemented for GPUs. The first optimization fuses al”
All SMILES Variational Autoencoder (2019, background): “LSTMs and GRUs are ubiquitous in natural language processing tasks, and efﬁcient GPU implementations have been developed [1].”
A Lightweight Recurrent Network for Sequence Modeling (2019, background): “Appleyard et al. (2016) upgrade NIVIDIA’s cuDNN implementation through exposing parallelism between operations within the recurrence.”
A Scalable Near-Memory Architecture for Training Deep Neural Networks on Large In-Memory Datasets (2018, methodology): “Until additional light is shed on the topic, a training accelerator must support 32 bit FP arithmetic to be able to compete with the ubiquitous GPU.”
Training RNNs as Fast as CNNs (2017, methodology): “In contrast, convolution conv2d and cuDNN LSTM (Appleyard et al., 2016) have been optimized as CUDA kernel functions to accelerate their computation.”
Simple Recurrent Units for Highly Parallelizable Recurrence (2017, methodology): “Our implementation for SRU is inspired by the cuDNN-optimized LSTM (Appleyard et al., 2016), but enables more parallelism – while cuDNN LSTM requires six optimization steps, SRU achieves more signiﬁcant speed-up via two optimizations.”