Generative Pretraining From Pixels

By Mark Chen, Alec Radford, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, D. Luan, Ilya Sutskever. 2020

1,828 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “…for object detection (Hu et al., 2018; Carion et al., 2020), video processing (Wang et al., 2018; Sun et al., 2019), image classiﬁcation (Wu et al., 2020), unsupervised object discovery (Locatello et al., 2020), or uniﬁed text-vision tasks (Chen et al., 2020c; Lu et al., 2019; Li et al., 2019).”
Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “SimCLRv2 The SimCLRv2 (Chen et al., 2020d) project released pre-trained and fine-tuned models in various settings.”
Masked Autoencoders Are Scalable Vision Learners (2021, methodology): “Motivated by the success in NLP, related recent methods [6, 16, 2] are based on Transformers [57]. iGPT [6] operates on sequences of pixels and predicts unknown pixels.”
High-Resolution Image Synthesis with Latent Diffusion Models (2021, background): “While autoregressive models (ARM) [6, 10, 91, 92] achieve strong performance in density estimation, computationally demanding architectures [94] and a sequential sampling process limit them to low resolution images.”
Evaluating Large Language Models Trained on Code (2021, background): “…& Le, 2015; Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018), computer vision (Van Oord et al., 2016; Menick & Kalchbrenner, 2018; Chen et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al., 2016; 2018; Dhari-wal et al., 2020; Baevski et al., 2020), biology…”
Early Convolutions Help Transformers See Better (2021, background): “, tuning AdamW’s β2 can be important [3]) and requiring 50% less optimizer state memory, which can ease scaling.”
VOLO: Vision Outlooker for Visual Recognition (2021, background): “including image classification [2], [3], [4], object detection [5], [6] and image processing like denoising, super-resolution and deraining [7].”
An Empirical Study of Training Self-Supervised Vision Transformers (2021, methodology): “prior works [8, 15] that train self-supervised Transformers with masked auto-encoding, we study the frameworks that are based on Siamese networks, including MoCo [19] and others [9, 17, 7].”
Zero-Shot Text-to-Image Generation (2021, background): “, 2019), images (Chen et al., 2020), and audio (Dhariwal et al.”
Training data-efficient image transformers & distillation through attention (2020, background): “Despite several attempts to use transformers for image classification [7], until now their performance has been inferior to that of convnets.”
Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “Other likelihood-based models include autoregressive models [8, 13,41,43] and diffusion models [11,20,21].”
Vector-quantized Image Modeling with Improved VQGAN (2021, methodology): “Similar to findings reported in iGPT (Chen et al., 2020a), the representations from the middle transformer blocks has better linear-probe accuracy (more study can be found in Appendix).”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “[13, 48]), but the limited scalability of Transformers restricts their usefulness as general-purpose architectures.”
CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders (2021, neutral): “While GAN-based methods have enabled considerable progress towards photorealistic image synthesis, strong autoregressive models have achieved similar quality results (Oord et al., 2017; Chen et al., 2020), with the recent DALL-E model (Ramesh et al.”
CogView: Mastering Text-to-Image Generation via Transformers (2021, methodology): “Even the biggest pixel-level auto-regressive model, ImageGPT [7], was pretrained on ImageNet at a max resolution of only 96 × 96 .”
Diffusion Models Beat GANs on Image Synthesis (2021, background): “These models can be used in a variety of ways, such as generating images from text prompts [78, 56] or learning useful feature representations [18, 10].”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, positive): “These latents are then modeled using a strong autoregressive prior using a GPTlike (Radford et al., 2019; Child et al., 2019; Chen et al., 2020) architecture.”
Perceiver: General Perception with Iterative Attention (2021, methodology): “A variety of strategies have been proposed to reduce the size of the input to the Transformer so it can be used on domains that are otherwise too large, including subsampling the input (Chen et al., 2020a) or by ﬁrst preprocessing the input using convolutions (e.g. Wu et al. 2020).”