Big Bidirectional Insertion Representations for Documents
By Lala Li, William Chan. 2019
4 citations recorded by the source corpus.
The Insertion Transformer is well suited for long form text generation due to its parallel generation capabilities, requiring O(\log_2 n) generation steps to generate n tokens. However, modeling long sequences is difficult, as there is more ambiguity captured in the attention mechanism. This work proposes the Big Bidirectional Insertion Representations for Documents (Big BIRD), an insertion-based model for document-level translation tasks. We scale up the insertion-based models to long form documents. Our key contribution is introducing sentence alignment via sentence-positional embeddings between the source and target document. We show an improvement of +4.3 BLEU on the WMT’19 English->German document-level translation task compared with the Insertion Transformer baseline.
Subfields: Artificial Intelligence, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yu Zhang
- Mohammad Norouzi
- William Chan
- Quoc V. Le
- Ron J. Weiss
- Yoshua Bengio
- Shinji Watanabe
- Jinyu Li
In Model Architectures
- Mohammad Norouzi
- Yu Zhang
- William Chan
- Navdeep Jaitly
- Ron J. Weiss
- Yoshua Bengio
- Chitwan Saharia
- Geoffrey E. Hinton
Citations in the corpus
- Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, methodology): “The Insertion Transformer (Stern et al., 2019) and other insertion-based models (Gu et al., 2019a; Welleck et al., 2019; Chan et al., 2019b; Li & Chan, 2019; Ruis et al., 2019) generates sequences through a series of insertions.”
- WaveGrad: Estimating Gradients for Waveform Generation (2020, background): “…Ghazvininejad et al., 2019; Chan et al., 2020; Saharia et al., 2020), insertion-based generative models (Stern et al., 2019; Chan et al., 2019b;a;c; Li & Chan, 2019) and edit-based generative models (Sabour et al., 2019; Gu et al., 2019; Ruis et al., 2019) found in the semi-autoregressive sequence…”
- Non-Autoregressive Machine Translation with Latent Alignments (2020, methodology): “, 2019, 2020b), while others require a logarithmic number of generation steps (Stern et al., 2019; Chan et al., 2019b,a; Li and Chan, 2019).”