Argmax Flows and Multinomial Diffusion: Towards Non-Autoregressive Language Models

By Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forr'e, Max Welling. 2021

79 citations recorded by the source corpus.

Subfields: Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Generative Models

Citations in the corpus

On Fast Sampling of Diffusion Probabilistic Models (2021, background): “D point cloud generation (Luo and Hu, 2021; Zhou et al., 2021), and language models (Hoogeboom et al., 2021).”
Noise Estimation for Generative Diffusion Models (2021, background): “In another concurrent work, a multinomial diffusion process is learned by adding categorical noise to the process [9].”
Vector Quantized Diffusion Model for Text-to-Image Synthesis (2021, methodology): “While the majority of recent works focus on continuous diffusion models, using them for categorical distribution is much less researched [1, 23].”
Palette: Image-to-Image Diffusion Models (2021, background): “…models [Ho et al. 2020; Sohl-Dickstein et al. 2015; Song and Ermon 2020] have received a surge of recent interest [Austin et al. 2021; Cai et al. 2020; Hoogeboom et al. 2021; Kingma et al. 2021; Song et al. 2021; Vahdat et al. 2021], resulting in several key advances in modeling continuous data.”
Diffsound: Discrete Diffusion Model for Text-to-Sound Generation (2022, background): “[38] extended the model to categorical random variables with transition matrices characterized by uniform transition probabilities.”
InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, background): “[45] extend the model to categorical random variables with transition matrices characterized by uniform transition probabilities.”
A Variational Perspective on Diffusion-Based Generative Models and Score Matching (2021, methodology): “This line of work has been successfully applied to modeling high dimensional natural images (Dhariwal & Nichol, 2021; Saharia et al., 2021), audio (Kong et al., 2020), 3D point cloud (Cai et al., 2020; Zhou et al., 2021), and discrete data (Hoogeboom et al., 2021).”
Structured Denoising Diffusion Models in Discrete State-Spaces (2021, methodology): “[18], by going beyond corruption processes with uniform transition probabilities.”
SegDiff: Image Segmentation with Diffusion Probabilistic Models (2021, methodology): “Diffusion models were also applied to language modeling [2, 20], where a novel diffusion model for categorical data was used.”
Step-unrolled Denoising Autoencoders for Text Generation (2021, background): “More details and relations to the forward diffusion process (Hoogeboom et al., 2021) are presented in Appendix A.3.”
Itô-Taylor Sampling Scheme for Denoising Diffusion Probabilistic Models using Ideal Derivatives (2021, background): “, 2021), natural language generation (Hoogeboom et al., 2021; Austin et al., 2021), etc.”
Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling (2021, background): “…Jolicoeur-Martineau et al. (2021b); Ho et al. (2020); Song and Ermon (2020); Song et al. (2020, 2021); Niu et al. (2020); Durkan and Song (2021); Hoogeboom et al. (2021); Saharia et al. (2021); Luhman and Luhman (2021, 2020); Nichol and Dhariwal (2021); Popov et al. (2021); Dhariwal and Nichol…”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “They generalize the multinomial diffusion model [85] through going beyond corruption processes with uniform transition probabilities.”
NExT-GPT: Any-to-Any Multimodal LLM (2023, methodology): “Moreover, to generate high-quality content, a multitude of strong-performing methods have been proposed, such as Transformer [82, 101, 17, 24], GANs [53, 7, 93, 110], VAEs [81, 67], Flow models [73, 6] and the current state-of-the-art diffusion models [31, 64, 57, 22, 68].”
Zero-Shot Translation using Diffusion Models (2021, methodology): “We use a neural network µ which predicts a probability vector at each diffusion step, similar to the method used by Hoogeboom et al. [13].”
Pseudo Numerical Methods for Diffusion Models on Manifolds (2022, methodology): “DDPMs have been applied successfully to a variety of applications, including image generation (Ho et al., 2020; Song et al., 2020b), text generation (Hoogeboom et al., 2021; Austin et al., 2021), 3D point cloud generation (Luo & Hu, 2021), text-to-speech (Kong et al., 2021; Chen et al., 2020) and…”