Blank Language Models

By Tianxiao Shen, Victor Quach, Regina Barzilay, Tommi S. Jaakkola. 2020

77 citations recorded by the source corpus.

We propose Blank Language Model (BLM), a model that generates sequences by dynamically creating and filling in blanks. The blanks control which part of the sequence to expand, making BLM ideal for a variety of text editing and rewriting tasks. The model can start from a single blank or partially completed text with blanks at specified locations. It iteratively determines which word to place in a blank and whether to insert new blanks, and stops generating when no blanks are left to fill. BLM can be efficiently trained using a lower bound of the marginal data likelihood. On the task of filling missing text snippets, BLM significantly outperforms all other baselines in terms of both accuracy and fluency. Experiments on style transfer and damaged ancient text restoration demonstrate the potential of this framework for a wide range of applications.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Natural Language Processing

Citations in the corpus

Augmented Language Models: a Survey (2023, background): “Other approaches that teach models to iteratively improve texts in an unsupervised fashion range from applications such as blank filling (Shen et al., 2020; Donahue et al., 2020) to denoising a sequence of Gaussian vectors into word vectors (Li et al.”
PEER: A Collaborative Language Model (2022, background): “Various approaches teach models to iteratively improve texts in an unsupervised fashion (e.g., Shen et al., 2020; Donahue et al., 2020; Li et al., 2022) and explore more efficient ways of representing edits (Mallinson et al., 2020).”
Pre-Trained Models: Past, Present and Future (2021, methodology): “Both Q-BERT and TernaryBERT result in ultra low-bit models.”
Efficient Training of Language Models to Fill in the Middle (2022, methodology): “Similarly, Blank Language models [Shen et al., 2020] generate text by iteratively selecting a blank and replacing it with a token (and optionally more blanks).”
GLM: General Language Model Pretraining with Autoregressive Blank Infilling (2021, methodology): “, 2017) and compare it with Blank Language Model (BLM) (Shen et al., 2020), which is a specifically designed model for text infilling.”
Enabling Language Models to Fill in the Blanks (2020, background): “Zhu et al. (2019); Shen et al. (2020) in-ﬁll multiple variable-length sequences, but these approaches require the masked context to be iteratively updated and reprocessed to ﬁll in blanks one a time.”
DeepStruct: Pretraining of Language Models for Structure Prediction (2022, background): “These models have beneﬁted a wide range of nature language generation tasks such as summarization (Zhang et al., 2020) and text inﬁlling (Zhu et al., 2019; Shen et al., 2020).”
Progressive Generation of Long Text with Pretrained Language Models (2021, unknown): “…generation (Welleck et al., 2019; Gu et al., 2019; Stern et al., 2019; Chan et al., 2019; Zhang et al., 2020), inﬁlling (Zhu et al., 2019; Shen et al., 2020; Qin et al., 2020), or reﬁnement (Lee et al., 2018; Novak et al., 2016; Mansimov et al., 2019; Kasai et al., 2020) that differs…”
FELIX: Flexible Text Editing Through Tagging and Insertion (2020, background): “Instead of the left-to-right autoregressive decoding, Insertion Transformer (Stern et al., 2019) and BLM (Shen et al., 2020) generate the output sequence through insertion operations, whereas Lev-enshtein Transformer (L EV T) (Gu et al., 2019) additionally incorporates a deletion operation.”