Step-unrolled Denoising Autoencoders for Text Generation

By Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, Aäron van den Oord. 2021

152 citations recorded by the source corpus.

In this paper we propose a new generative model of text, Step-unrolled Denoising Autoencoder (SUNDAE), that does not rely on autoregressive models. Similarly to denoising diffusion techniques, SUNDAE is repeatedly applied on a sequence of tokens, starting from random inputs and improving them each time until convergence. We present a simple new improvement operator that converges in fewer iterations than diffusion methods, while qualitatively producing better samples on natural language datasets. SUNDAE achieves state-of-the-art results (among non-autoregressive methods) on the WMT'14 English-to-German translation task and good qualitative results on unconditional language modeling on the Colossal Cleaned Common Crawl dataset and a dataset of Python code from GitHub. The non-autoregressive nature of SUNDAE opens up possibilities beyond left-to-right prompted generation, by filling in arbitrary blank patterns in a template.

Subfields: Generative Models, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

1. David J. Fleet
2. Ting-Li Chen
3. Arnaud Doucet
4. Kihyuk Sohn
5. Yilun Du
6. Allan Jabri
7. Will Grathwohl
8. Lu Jiang

In Natural Language Generation

1. Ting-Li Chen
2. Xin Sun
3. Ming-Hsuan Yang
4. Heming Xia
5. Tal Remez
6. Furu Wei
7. Tao Ge

Citations in the corpus

• Scalable Adaptive Computation for Iterative Generation (2022, methodology): “It bears similarity to stepunrolled autoencoders (Savinov et al., 2021) and has been adopted in several existing work (Strudel et al.”
• Photorealistic Video Generation with Diffusion Models (2023, background): “In addition to being conditioned on external inputs, iterative generative algorithms can also be conditioned on their own previously generated samples during inference [3, 13, 65].”
• DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models (2022, background): “There are other works of diffusion on text generation, but they stick to the original encoderdecoder architecture and the diffusion process is interspersed on the decoder (Savinov et al., 2021), or the latent space (Yu et al.”
• Discrete Flow Matching (2024, methodology): “Savinov et al. (2021) extended the mask-modeling approach by employing an additional loss term that incorporates rolling model predictions.”
• A Continuous Time Framework for Discrete Denoising Models (2022, background): “[26] also focus on text, creating a generative reverse chain that repeatedly applies the same denoising kernel.”
• Continuous diffusion for categorical data (2022, methodology): “This approach bears a strong resemblance to the unrolled denoising strategy proposed by Savinov et al. (2021).”
• Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning (2022, background): “Other related work The proposed Self-Conditioning technique shares some similarities with selfmodulation in GANs [7] (where the earlier latent state can directly modulate the later latent states) and step-unrolled denoising autoencoders [48] (where an inference step is incorporated for denoising).”
• Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “Recent advances in deep learning have facilitated significant progress for various data modalities [25, 162, 207], such as images [34, 237], audio [112, 236], and text [12, 18, 80, 160, 163].”
• Self-conditioned Embedding Diffusion for Text Generation (2022, methodology): “SUNDAE (Savinov et al., 2022) was the first non-AR method to show strong results both in machine translation and unconditional text generation.”
• Lossless Acceleration for Seq2seq Generation with Aggressive Decoding (2022, methodology): “…3.2.1 and 3.2.2 for the drafter and verifier using a shared teacher: Following recent NAR work (Ghazvininejad et al., 2019; Saharia et al., 2020; Savinov et al., 2021), we use the Transformerbig as the teacher for WMT14 EN↔DE, and use Transformer-base for WMT16 EN↔RO, which all train with the…”
• Broadly applicable and accurate protein design by integrating structure prediction networks and diffusion generative models (2022, unknown): “Self-conditioning is closely related to Step Unrolled Denoising Autoencoders 52 ; both methods perform a step of inference at training time to train the model to expect inputs of the same distribution that will be generated at inference time.”
• Simplified and Generalized Masked Diffusion for Discrete Data (2024, unknown): “SUNDAE [51, 71].”