Fast Generation for Convolutional Autoregressive Models

By Prajit Ramachandran, T. Paine, Pooya Khorrami, Mohammad Babaeizadeh, Shiyu Chang, Yang Zhang, Mark Hasegawa-Johnson, Roy H. Campbell, Thomas S. Huang. 2017

67 citations recorded by the source corpus.

Convolutional autoregressive models have recently demonstrated state-of-the-art performance on a number of generation tasks. While fast, parallel training methods have been crucial for their success, generation is typically implemented in a naive fashion where redundant computations are unnecessarily repeated. This results in slow generation, making such models infeasible for production environments. In this work, we describe a method to speed up generation in convolutional autoregressive models. The key idea is to cache hidden states to avoid redundant computation. We apply our fast generation method to the Wavenet and PixelCNN++ models and achieve up to $21\times$ and $183\times$ speedups respectively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Predictive Sampling with Forecasting Autoregressive Models (2020, methodology): “For example, (Ramachandran et al., 2017) report a speedup of 2 × for batch size 1 and 33 × for batch size 32 for Pixel-CNN++ models trained on CIFAR10.”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, methodology): “(Ramachandran et al., 2017).”
Parallel Multiscale Autoregressive Density Estimation (2017, methodology): “Causally-structured convolutional networks such as Pixel-CNN (van den Oord et al., 2016b) and WaveNet (Oord et al., 2016) improved the speed and scalability of training.”
PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples (2017, methodology): “Nonetheless, by caching redundant calculation, Ramachandran et al. (2017) proposes a very fast generation algorithm for PixelCNN.”
Conditional Time Series Forecasting with Convolutional Neural Networks (2017, methodology): “…for longer time series the prediction of the autoregressive model may be sped up by implementing a recent variation that exploits the memorization structure of the network, see [21] or speeding up the convolutions by working in the frequency domain emloying Fourier transforms as in [19], [22].”
Pixel Recursive Super Resolution (2017, background): “Recent work [34] has shown that caching intermediate activation states can improve sample time dramatically.”
Efficiently Modeling Long Sequences with Structured State Spaces (2021, methodology): “For the PixelCNN++, we used the fast cached version by Ramachandran et al. [34], which sped up generation by orders of magnitude from the naive implementation.”
HiLLoC: Lossless Image Compression with Hierarchical Latent Variable Models (2019, background): “This effect can be mitigated by using dynamic programming (Le Paine et al., 2016; Ramachandran et al., 2017), and altering model architecture (Reed et al.”
GELP: GAN-Excited Liner Prediction for Speech Synthesis from Mel-spectrogram (2019, background): “Although the inference can be made faster with dilation buffering [10], or optimizing recurrent neural networks (RNNs) for real-time use [11], sequential inference remains ill-suited for massively parallel applications.”
An Investigation of Subband Wavenet Vocoder Covering Entire Audible Frequency Range with Limited Acoustic Features (2018, methodology): “In the synthesis, a fast generation algorithm was introduced [29].”
VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation (2019, methodology): “Prior work has aimed to speed up training and synthesis with such auto-regressive models (Reed et al., 2017; Ramachandran et al., 2017).”
Latent Variable Models (2019, background): “This effect can be mitigated by using dynamic programming (Le Paine et al., 2016; Ramachandran et al., 2017), and altering model architecture (Reed et al.”
GELP: GAN-Excited Linear Prediction for Speech Synthesis from Mel-spectrogram (2019, background): “Although the inference can be made faster with dilation buffering [10], or optimizing recurrent neural networks (RNNs) for real-time use [11], sequential inference remains ill-suited for massively parallel applications.”
Dilated convolutional neural networks for time series forecasting (2018, methodology): “…for longer time series the prediction of the autoregressive model may be sped up by implementing a recent variation that exploits the memorization structure of the network, see [21] or speeding up the convolutions by working in the frequency domain emloying Fourier transforms as in [19], [22].”
PixColor: Pixel Recursive Colorization (2017, methodology): “In addition, there are various additional speedup tricks we can use (see e.g., [27, 19]) if necessary.”
Deep autoregressive models for the efficient variational simulation of many-body quantum systems (2019, background): “Carefully exploiting the intrinsic sparseness of the network weights, further leads to a very efficient algorithm for sampling [19].”
PixelCNN Models with Auxiliary Variables for Natural Image Modeling (2016, background): “However, as reported in (Ramachandran et al., 2017), caching delivers minor sampling speed gain, when only a few or a single image is generated at once, which is a common scenario in real-life applications.”
Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving (2020, methodology): “Under certain cases, the computation for h t ( u , s 1: t − 1 ) can be cached to save the time for computing h t +1 ( u , s 1: t ) ( cf ., (Ramachandran et al., 2017) for autoregressive models).”
VideoFlow: A Flow-Based Generative Model for Video (2019, methodology): “Prior work has aimed to speed up training and synthesis with such auto-regressive models (Reed et al., 2017; Ramachandran et al., 2017).”