MaskGAN: Better Text Generation via Filling in the ______

By William Fedus, Ian J. Goodfellow, Andrew M. Dai. 2018

489 citations recorded by the source corpus.

Neural text generation models are often autoregressive language models or seq2seq models. These models generate text by sampling words sequentially, with each word conditioned on the previous word, and are state-of-the-art for several machine translation and summarization benchmarks. These benchmarks are often defined by validation perplexity even though this is not a direct measure of the quality of the generated text. Additionally, these models are typically trained via maxi- mum likelihood and teacher forcing. These methods are well-suited to optimizing perplexity but can result in poor sample quality since generating text requires conditioning on sequences of words that may have never been observed at training time. We propose to improve sample quality using Generative Adversarial Networks (GANs), which explicitly train the generator to produce high quality samples and have shown a lot of success in image generation. GANs were originally designed to output differentiable values, so discrete language generation is challenging for them. We claim that validation perplexity alone is not indicative of the quality of text generated by a model. We introduce an actor-critic conditional GAN that fills in missing text conditioned on the surrounding context. We show qualitatively and quantitatively, evidence that this produces more realistic conditional and unconditional text samples compared to a maximum likelihood trained model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Generative Models

Citations in the corpus

XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, background): “Another related idea is to perform autoregressive denoising in the context of text generation [11], which only considers a fixed order though.”
MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, background): “…et al., 2018; Hassan et al., 2018; Song et al., 2018; Shen et al., 2018), text summarization (Ayana et al., 2016; Suzuki & Nagata, 2017; Gehring et al., 2017), question answering (Yuan et al., 2017; Fedus et al., 2018) and conversational response generation (Shang et al., 2015; Vinyals & Le, 2015).”
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, methodology): “We pre-train encoder-decoder models using the span-based masked language modeling (MLM) objective (Fedus et al., 2018; Devlin et al., 2018).”
ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, background): “…provided by the discriminator (see Section 3.2), a large body of works focused on deﬁning denser rewards: ranking and comparative discriminators [20, 59], sequential discriminators where the rewards are provided at each time step of the generation [39, 23], or using masked language modeling [11].”
Generative Adversarial Networks (2021, neutral): “By providing a thorough overview of GANs, we hope to make this complex and challenging topic more accessible to researchers and practitioners in the field, and inspire new innovations and applications.”
GeDi: Generative Discriminator Guided Sequence Generation (2020, background): “, 2014) in the language generation domain (Yu et al., 2017; Fedus et al., 2018).”
Towards Coherent and Cohesive Long-form Text Generation (2018, background): “Another line of research involves the use of Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) to incorporate feedback signals for text generation (Yu et al., 2017; Lin et al., 2017; Zhang et al., 2017; Guo et al., 2017; Fedus et al., 2018; Zhang et al., 2018).”
Generative adversarial networks (2014, neutral): “GANs struggle to generate discrete data because the back-propagation algorithm needs to propagate gradients from the discriminator through the output of the generator, but this problem is being gradually resolved.(9) Like most generative models, GANs can be used to fill in gaps in missing data.”
Do Transformer Modifications Transfer Across Implementations and Applications? (2021, methodology): “We pre-train encoder-decoder models in a self-supervised manner using the “span corruption” masked language modeling objective (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) on the C4 dataset.”
Synthesizing Tabular Data using Generative Adversarial Networks (2018, background): “However, generating discrete variables is a challenge for GANs: the authors in [26, 7] attempt a differential model by designing special functions or modifying the loss function; other researchers [50, 14, 46] use a reinforcement learning framework to train the non-differentiable model, making natur”
GAN Memory with No Forgetting (2020, background): “Generative adversarial networks (GANs) GANs continue to show increasing power in synthesizing highly realistic observations [32, 45, 51, 9, 33, 34], and have found wide applicability in various ﬁelds [39, 1, 19, 79, 81, 82, 12, 87, 37].”
Efficient Training of Language Models to Fill in the Middle (2022, methodology): “Text infilling can also be performed using a GAN [Fedus et al., 2018], but REINFORCE is required to deal with the discreteness of text.”
Language GANs Falling Short (2018, background): “, 2017), step-level loss instead of sequence level (Fedus et al., 2018), learned baseline to reduce variance (Fedus et al.”
Improving Abstraction in Text Summarization (2018, background): “A recent approach to training sequential models utilizes generative adversarial networks to improving the human perceived quality of generated outputs (Fedus et al., 2018; Guimaraes et al., 2017; Liu et al., 2018).”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “GANs have already been used successfully in a variety of applications, including image captioning[26, 126], image to image translation [56], text to image synthesis [15, 115], and text generation [36, 145].”
Enabling Language Models to Fill in the Blanks (2020, methodology): “In contrast, using LMs to directly predict x from ˜ x as in Fedus et al. (2018) effectively doubles the sequence length of x .”
Blank Language Models (2020, background): “Additional insertion control is provided by the masked language model where each mask corresponds to a single word (Fedus et al., 2018).”