Enabling Language Models to Fill in the Blanks

By Chris Donahue, Mina Lee, Percy Liang. 2020

212 citations recorded by the source corpus.

We present a simple approach for text infilling, the task of predicting missing spans of text at any position in a document. While infilling could enable rich functionality especially for writing assistance tools, more attention has been devoted to language modeling—a special case of infilling where text is predicted at the end of a document. In this paper, we aim to extend the capabilities of language models (LMs) to the more general task of infilling. To this end, we train (or fine tune) off-the-shelf LMs on sequences containing the concatenation of artificially-masked text and the text which was masked. We show that this approach, which we call infilling by language modeling, can enable LMs to infill entire sentences effectively on three different domains: short stories, scientific abstracts, and lyrics. Furthermore, we show that humans have difficulty identifying sentences infilled by our approach as machine-generated in the domain of short stories.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

InCoder: A Generative Model for Code Infilling and Synthesis (2022, background): “[23] and CM3 [2], who train left-to-right language models to fill in masked token segments of varying lengths; and the work of Alon et al.”
Augmented Language Models: a Survey (2023, background): “Other approaches that teach models to iteratively improve texts in an unsupervised fashion range from applications such as blank filling (Shen et al., 2020; Donahue et al., 2020) to denoising a sequence of Gaussian vectors into word vectors (Li et al.”
PEER: A Collaborative Language Model (2022, background): “Various approaches teach models to iteratively improve texts in an unsupervised fashion (e.g., Shen et al., 2020; Donahue et al., 2020; Li et al., 2022) and explore more efficient ways of representing edits (Mallinson et al.”
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models (2021, methodology): “To allow for targeted counterfactuals, we also design control codes like negation or delete (Figure 1B), and adopt fill-in-the-blank structures (Donahue et al., 2020) to specify where the perturbation occurs and how.”
GLM: General Language Model Pretraining with Autoregressive Blank Infilling (2021, background): “Text inﬁlling is the task of predicting missing spans of text which are consistent with the surrounding context (Zhu et al., 2019; Donahue et al., 2020; Shen et al., 2020).”
Blank Language Models (2020, methodology): “• Seq2seq-full (Donahue et al., 2020): We train a seq2seq model to output the full document x from input c.”
Simple is Better! Lightweight Data Augmentation for Low Resource Slot Filling and Intent Classification (2020, methodology): “However, we do not use D to look for substitute candidates, instead we use a large pre-trained language model to generate the slot value candidates, using the fill-in-the-blank style (Donahue et al., 2020).”
Self-Training: A Survey (2022, background): “Common approaches in self-supervised learning include predicting missing parts of an image [46], predicting the order of shuffled image patches or their orientation [73], reconstructing corrupted images [25], filling in missing words in a sentence [21], or predicting future frames in a video sequenc”
InFillmore: Frame-Guided Language Generation with Bidirectional Context (2021, methodology): “Akin to Donahue et al. (2020), we sampled 100 stories from the test set of the ROCStories dataset.”
Controlling Text Edition by Changing Answers of Specific Questions (2021, methodology): “For example, in the method using global context (Donahue et al., 2020), it is an pretrained language model by itself.”
Automated Storytelling via Causal, Commonsense Plot Ordering (2020, methodology): “Our problem setting requires us to generate a section of text between two consecutive high level plot points at a time, reminiscent of approaches taken by Ippolito et al. (2019) and Donahue, Lee, and Liang (2020) that condition a language model on left and right contexts to ﬁll in blanks in a story.”
Help me write a Poem - Instruction Tuning as a Vehicle for Collaborative Poetry Writing (2022, background): “, 2022; Branwen, 2020) such as building co-creative assistants for writing stories, poems, and argumentative essays (Lee et al., 2022; Swanson et al., 2021; Uthus et al., 2019; Donahue et al., 2020; Padmakumar and He, 2022; Du et al., 2022).”
Goal-Directed Story Generation: Augmenting Generative Language Models with Reinforcement Learning (2021, background): “One means of controlling story generation is to condition generation on high-level plot outlines (Fan et al., 2018; Peng et al., 2018; Rashkin et al., 2020) or story in-filling (Donahue et al., 2020).”
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning (2020, methodology): “To choose the best output across iterations, we employ an unsupervised ranking step based on BERT’s next sentence prediction task to measure coherence (Devlin et al., 2018).”