Distribution Augmentation for Generative Modeling

By Heewoo Jun, Rewon Child, Mark Chen, John Schulman, Aditya Ramesh, Alec Radford, Ilya Sutskever. 2020

70 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

Citations in the corpus

Variational Diffusion Models (2021, background): “43 Sparse Transformer + DistAug [Jun et al., 2020] AR 2.”
Score-based Generative Modeling in Latent Space (2021, background): “[78] Heewoo Jun, Rewon Child, Mark Chen, John Schulman, Aditya Ramesh, Alec Radford, and Ilya Sutskever.”
Efficient Training of Language Models to Fill in the Middle (2022, background): “DistAug [Jun et al., 2020] is another related work that trains jointly on transformed data while conditioning on the transformation type.”
Learning Energy-Based Models by Diffusion Recovery Likelihood (2020, neutral): “Model BPD ↓ DDPM (Ho et al., 2020) 3.70 Glow (Kingma & Dhariwal, 2018) 3.35 Flow++ (Ho et al., 2019) 3.08 PixelCNN (Van den Oord et al., 2016) 3.03 Sparse Transformer (Child et al., 2019) 2.80 DistAug (Jun et al., 2020) 2.56 Ours † ( T1k ) 3.18 Interpolation.”
Maximum Likelihood Training of Score-Based Diffusion Models (2021, background): “The likelihood for CIFAR-10 can be significantly improved by incorporating advanced data augmentation, as demonstrated in [25, 41].”
Learning to Efficiently Sample from Diffusion Probabilistic Models (2021, background): “…[2020], Nichol and Dhariwal [2021], augmenting UNet with self-attention [Shaw et al., 2018] in scales where attention is computationally feasible has helped bring DDPMs closer to the current state-of-the-art autoregressive generative models [Child et al., 2019, Jun et al., 2020, Roy et al., 2021].”
Score Matching Model for Unbounded Data Score (2021, background): “53 [20]) on CIFAR10 with performance of 2.”
Text Data Augmentation for Deep Learning (2021, methodology): “Further, DistAug [138] even tests Data Augmentation with large scale pixel autoregressive modeling in the ImageGPT model [139].”
SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis (2023, methodology): “This is a form of conditioning-augmentation, and has been used in various forms with autoregressive [20] models, and more recently with diffusion models [21].”
Graph Contrastive Learning Automated (2021, background): “That aligns with the observations in (Lee et al., 2020; Jun et al., 2020), showing that disentan-gling augmented and original feature distributions could have the model beneﬁt more from stronger augmentations.”
Combiner: Full Attention Transformer with Sparse Computation Cost (2021, methodology): “We run all of the models in Table 2 without data augmentation [33].”
iFlow: Numerically Invertible Flows for Efficient Lossless Compression via a Uniform Coder (2021, background): “Compared to other classes of generative models, normalizing flows typically perform best in both tasks of probability density estimation (as compared with variational autoencoders [24, 15, 8]) and inference speed (as compared with autoregressive factorizations [35, 39, 22]).”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, unknown): “+ DistAug [32], [110] ????? ????? ????? ????? 7 3 14.”
Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning (2022, methodology): “…are based on autoregressive modeling (Van den Oord et al., 2016; Salimans et al., 2017; Parmar et al., 2018; Child et al., 2019; Roy et al., 2021; Jun et al., 2020; Sutskever et al., 2014; Brown et al., 2020; Chowdhery et al., 2022), where the networks, often Transformers (Vaswani et al., 2017),…”
Distribution Augmentation for Low-Resource Expressive Text-To-Speech (2022, unknown): “As explained in [7], DistAug method can be interpreted as a form of data-dependent regularization that reduces overfitting.”
Efficient-VDVAE: Less is more (2022, background): “Deep autoregressive models[1, 2, 3, 4] have long achieved the best log likelihoods across different modalities.”