Parallel and Flexible Sampling from Autoregressive Models via Langevin Dynamics

By Vivek Jayaram, John Thickstun. 2021

27 citations recorded by the source corpus.

This paper introduces an alternative approach to sampling from autoregressive models. Autoregressive models are typically sampled sequentially, according to the transition dynamics defined by the model. Instead, we propose a sampling procedure that initializes a sequence with white noise and follows a Markov chain defined by Langevin dynamics on the global log-likelihood of the sequence. This approach parallelizes the sampling process and generalizes to conditional sampling. Using an autoregressive model as a Bayesian prior, we can steer the output of a generative model using a conditional likelihood or constraints. We apply these techniques to autoregressive models in the visual and audio domains, with competitive results for audio source separation, super-resolution, and inpainting.

Subfields: Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Signal Processing

Citations in the corpus

It's Raw! Audio Generation with State-Space Models (2022, methodology): “With AR models, computing the exact likelihood is tractable, which makes them simple to train, and lends them to applications such as lossless compression [22] and posterior sampling [19].”
ShapeFormer: Transformer-based Shape Completion via Sparse Representation (2022, background): “(iii) Adopt advanced sampling techniques for autoregressive models such as parallel sampling [8].”
Gradient-based Constrained Sampling from Language Models (2022, methodology): “…other gradient-based MCMC methods have been developed for generative modeling in continuous domains such as images (Song and Ermon, 2019) and audio (Jayaram and Thickstun, 2021) among others where the models are trained to predict the gradients (via a score function) directly whereas M U C O L A…”
Continuous diffusion for categorical data (2022, background): “, 2018) or alternative sampling algorithms (Jayaram and Thickstun, 2021; Song et al., 2021b), but these approaches have not been used for language models, to the best of our knowledge.”
COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics (2022, unknown): “Langevin dynamics is widely used on EBMs of modalities with continuous values, like images [48, 9, 59], 3D shapes [55], latent features [39], and audio sequences [22].”