KERMIT: Generative Insertion-Based Modeling for Sequences

By William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, Jakob Uszkoreit. 2019

67 citations recorded by the source corpus.

We present KERMIT, a simple insertion-based approach to generative modeling for sequences and sequence pairs. KERMIT models the joint distribution and its decompositions (i.e., marginals and conditionals) using a single neural network and, unlike much prior work, does not rely on a prespecified factorization of the data distribution. During training, one can feed KERMIT paired data $(x, y)$ to learn the joint distribution $p(x, y)$, and optionally mix in unpaired data $x$ or $y$ to refine the marginals $p(x)$ or $p(y)$. During inference, we have access to the conditionals $p(x \mid y)$ and $p(y \mid x)$ in both directions. We can also sample from the joint distribution or the marginals. The model supports both serial fully autoregressive decoding and parallel partially autoregressive decoding, with the latter exhibiting an empirically logarithmic runtime. We demonstrate through experiments in machine translation, representation learning, and zero-shot cloze question answering that our unified approach is capable of matching or exceeding the performance of dedicated state-of-the-art systems across a wide range of tasks without the need for problem-specific architectural adaptation.

Subfields: Artificial Intelligence, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

Citations in the corpus

RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “However, newer methods have improved performance by multi-task ﬁne tuning (Dong et al., 2019), incorporating entity embeddings (Sun et al., 2019), span prediction (Joshi et al., 2019), and multiple variants of autoregressive pretraining (Song et al., 2019; Chan et al., 2019; Yang et al., 2019).”
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, methodology): “While previous and concurrent work has looked at masking (Sun et al., 2019) or dropping (Song et al., 2019; Chan et al., 2019) multiple words from the input – particularly as pretraining for language generation tasks – SpanBERT pretrains span representations (Lee et al., 2016), which are widely…”
InCoder: A Generative Model for Code Infilling and Synthesis (2022, background): “Unfortunately, standard left-to-right language models cannot directly infill text, and popular masked language models are mainly trained to infill very short spans [17, 22, 52, 54].”
Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, methodology): “Such models sacriﬁce almost no performance, while requiring a logarithmic (Chan et al., 2019c) or a constant number of generation steps (Lee et al., 2018).”
WaveGrad: Estimating Gradients for Waveform Generation (2020, background): “…2019; Lee et al., 2018; Ghazvininejad et al., 2019; Chan et al., 2020; Saharia et al., 2020), insertion-based generative models (Stern et al., 2019; Chan et al., 2019b;a;c; Li & Chan, 2019) and edit-based generative models (Sabour et al., 2019; Gu et al., 2019; Ruis et al., 2019) found in the…”
Efficient Training of Language Models to Fill in the Middle (2022, background): “…the attention mask in a standard transformer to enable token generation in any userspecified order, while Insertion Transformer [Stern et al., 2019], KERMIT [Chan et al., 2019], and InDIGO [Gu et al., 2019] allow the model to predict a location for the next token before predicting the token.”
Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning (2020, methodology): “It’s reasonable to believe that these conclusions will generalize to other pre-trained language models such as Kermit (Chan et al., 2019), XLNet (Yang et al., 2019), GPT-2 (Radford et al., 2019), RoBERTa (Liu et al., 2019a) or ELMO (Peters et al., 2018).”
Blank Language Models (2020, methodology): “Training Previous work explored different strategies to design loss for training generalized sequence models (De-vlin et al., 2018; Yang et al., 2019; Stern et al., 2019; Chan et al., 2019).”
Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation (2020, unknown): “Non-autoregressive Translation In addition to the work already discussed, several other works proposed to iteratively reﬁne (or insert) output predictions (Mansimov et al., 2019; Stern et al., 2019; Gu et al., 2019a; Chan et al., 2019a,b; Li et al., 2020).”
Progressive Generation of Long Text with Pretrained Language Models (2021, methodology): “…yet those methods either require specialized neural architectures that need costly retraining for each domain (Gu et al., 2019; Stern et al., 2019; Chan et al., 2019; Fan et al., 2019), or rely on dedicated intermediate content plans (e.g., summaries, SRL labels) (Fan et al., 2019; Yao et al.,…”
Non-Autoregressive Machine Translation with Latent Alignments (2020, methodology): “, 2018), CTC models (Libovicky and Helcl, 2018), insertionbased methods (Stern et al., 2019; Chan et al., 2019b), edit-based methods (Gu et al.”
FELIX: Flexible Text Editing Through Tagging and Insertion (2020, methodology): “g, KERMIT (Chan et al., 2019), MASS (Song et al.”
SenseBERT: Driving Some Sense into BERT (2019, methodology): “Other proposed approaches for self-supervised objectives, including unidirectional [Radford et al., 2019], permutational [Yang et al., 2019], or word insertion-based [Chan et al., 2019] methods, operate similarly, over words.”
Insertion-Deletion Transformer (2020, background): “Recently, research in insertion-based non- or partially- autoregressive models has spiked (Stern et al., 2019; Welleck et al., 2019; Gu et al., 2019a; Chan et al., 2019).”
Teaching Machine Comprehension with Compositional Explanations (2020, background): “, 2020); (2) Learning Efficiency: a model learns quickly with minimal supervision (Radford et al., 2019; Chan et al., 2019); (3) Annotation Efficiency: to create a dataset efficiently with time limit/budget; our work falls into this category.”