Predictive Sampling with Forecasting Autoregressive Models

By Auke J. Wiggers, Emiel Hoogeboom. 2020

18 citations recorded by the source corpus.

Autoregressive models (ARMs) currently hold state-of-the-art performance in likelihood-based modeling of image and audio data. Generally, neural network based ARMs are designed to allow fast inference, but sampling from these models is impractically slow. In this paper, we introduce the predictive sampling algorithm: a procedure that exploits the fast inference property of ARMs in order to speed up sampling, while keeping the model intact. We propose two variations of predictive sampling, namely sampling with ARM fixed-point iteration and learned forecasting modules. Their effectiveness is demonstrated in two settings: i) explicit likelihood modeling on binary MNIST, SVHN and CIFAR10, and ii) discrete latent modeling in an autoencoder trained on SVHN, CIFAR10 and Imagenet32. Empirically, we show considerable improvements over baselines in number of ARM inference calls and sampling speed.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Stefano Ermon
2. Chenlin Meng
3. Yang Song
4. Sergey Levine
5. Renjie Liao
6. Pieter Abbeel
7. Geoffrey Irving
8. Mohammad Norouzi

In Computer Vision and Pattern Recognition

1. Stefano Ermon
2. Luc Van Gool
3. Yang Song
4. Chenlin Meng
5. Dacheng Tao
6. Ming-Hsuan Yang
7. Jitendra Malik
8. Bernt Schiele

In Generative Models

1. Yang Song
2. Stefano Ermon
3. Chenlin Meng
4. Arash Vahdat
5. Karsten Kreis
6. Jonathan Ho
7. Prafulla Dhariwal
8. Jiaming Song

In Signal Processing

1. Yang Song
2. Stefano Ermon
3. Chenlin Meng
4. Renjie Liao
5. Arash Vahdat
6. Jong Chul Ye
7. David J. Fleet
8. Prafulla Dhariwal

Citations in the corpus

• Denoising Diffusion Probabilistic Models (2020, neutral): “Our progressive decoding argument can be seen in convolutional DRAW and related models [16, 37] and may also lead to more general designs for subscale orderings or future predictions for autoregressive models [35, 61].”
• Accelerating Large Language Model Decoding with Speculative Sampling (2023, methodology): “This includes block parallel sampling (Stern et al., 2018), aggressive decoding (Ge et al., 2022), in addition to some work in parallelizing autoregressive models in the image domain (Song et al., 2021; Wiggers and Hoogeboom, 2020).”
• Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving (2020, background): “Concurrently, Wiggers & Hoogeboom (2020) propose to accelerate autoregressive sampling with a fixed-point iteration method and demonstrate advantages over feedforward sampling (without caching) on PixelCNN++ models.”