Insertion-Deletion Transformer

By Laura Ruis, Mitchell Stern, Julia Proskurnia, William Chan. 2020

14 citations recorded by the source corpus.

We propose the Insertion-Deletion Transformer, a novel transformer-based neural architecture and training method for sequence generation. The model consists of two phases that are executed iteratively, 1) an insertion phase and 2) a deletion phase. The insertion phase parameterizes a distribution of insertions on the current output hypothesis, while the deletion phase parameterizes a distribution of deletions over the current output hypothesis. The training method is a principled and simple algorithm, where the deletion model obtains its signal directly on-policy from the insertion model output. We demonstrate the effectiveness of our Insertion-Deletion Transformer on synthetic translation tasks, obtaining significant BLEU score improvement over an insertion-only model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yu Zhang
2. Mohammad Norouzi
3. Yoshua Bengio
4. Quoc V. Le
5. William Chan
6. Sergey Levine
7. Ron J. Weiss
8. Dong Yu

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Mohammad Norouzi
3. Ming-Hsuan Yang
4. Dacheng Tao
5. Trevor Darrell
6. Jitendra Malik
7. Xiaogang Wang
8. Andrew Zisserman

In Natural Language Processing

1. Yu Zhang
2. William Chan
3. Ron J. Weiss
4. Mohammad Norouzi
5. Navdeep Jaitly
6. Heiga Zen
7. Yoshua Bengio
8. Dong Yu

In Model Architectures

1. Mohammad Norouzi
2. Yu Zhang
3. William Chan
4. Navdeep Jaitly
5. Yoshua Bengio
6. Ron J. Weiss
7. Chitwan Saharia
8. Geoffrey E. Hinton

Citations in the corpus

• Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, methodology): “Insertion Transformer has been shown to generate sequences in log-arithmic log n iterations (Stern et al., 2019), where as the Imputer requires constant generation steps.”
• WaveGrad: Estimating Gradients for Waveform Generation (2020, background): “…2019; Chan et al., 2020; Saharia et al., 2020), insertion-based generative models (Stern et al., 2019; Chan et al., 2019b;a;c; Li & Chan, 2019) and edit-based generative models (Sabour et al., 2019; Gu et al., 2019; Ruis et al., 2019) found in the semi-autoregressive sequence generation literature.”
• Structured Denoising Diffusion Models in Discrete State-Spaces (2021, background): “Other works have considered non-autoregressive translation and speech transcription via insertion and deletion [16, 33], masking [13], and iteratively-refined sequence alignments [5, 34].”
• Non-Autoregressive Machine Translation with Latent Alignments (2020, methodology): “…refinement (Lee et al., 2018), CTC models (Libovicky and Helcl, 2018), insertion-based methods (Stern et al., 2019; Chan et al., 2019b), edit-based methods (Gu et al., 2019; Ruis et al., 2019), masked language models (Ghazvininejad et al., 2019, 2020b), and normalizing flow models (Ma et al., 2019).”