Constant-Time Machine Translation with Conditional Masked Language Models

By Marjan Ghazvininejad, Omer Levy, Yinhan Liu, Luke S. Zettlemoyer. 2019

35 citations recorded by the source corpus.

Most machine translation systems generate text autoregressively, by sequentially predicting tokens from left to right. We, instead, use a masked language modeling objective to train a model to predict any subset of the target words, conditioned on both the input text and a partially masked target translation. This approach allows for efficient iterative decoding, where we first predict all of the target words non-autoregressively, and then repeatedly mask out and regenerate the subset of words that the model is least confident about. By applying this strategy for a constant number of iterations, our model improves stateof-the-art performance levels for constant-time translation models by over 3 BLEU on average. It is also able to reach 92-95% of the performance of a typical left-to-right transformer model, while decoding significantly faster.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Defending Against Neural Fake News (2019, background): “This also suggests that recent progress on generating text in any order (Gu et al., 2019; Stern et al., 2019; Ghazvininejad et al., 2019) may lead to models that evade a Grover discriminator.”
Real or Fake? Learning to Discriminate Machine from Human Generated Text (2019, background): “906.03351v1 [cs.LG] 7 Jun 2019 2 Related Work While there is ongoing effort towards generating text without specifying a word ordering [Kaiser et al., 2018, Stern et al., 2018, 2019, Gu et al., 2019, Ghazvininejad et al., 2019], the dominant approach to text generation is still based on auto-regress”
Levenshtein Transformer (2019, methodology): “For fair comparison, we use the result with length beam 1 from Ghazvininejad et al. (2019).”
Blank Language Models (2020, methodology): “Finally, several works combine left-to-right language models with control codes or customized inference algorithms for more ﬂexible generation (Keskar et al., 2019; Sun et al., 2017; Liu et al., 2019).”
Understanding Knowledge Distillation in Non-autoregressive Machine Translation (2019, methodology): “• MaskPredict (MaskT, Ghazvininejad et al., 2019): MaskT adopts a masked language model (Devlin et al., 2018) to progressively generate the sequence from an entirely masked input.”
Imitation Attacks and Defenses for Black-box Machine Translation Systems (2020, unknown): “, 2019), and improving non-autoregressive generation (Ghazvininejad et al., 2019; Stern et al., 2019).”
A Survey of the Usages of Deep Learning for Natural Language Processing (2020, methodology): “[256] recently applied BERT to the machine translation task, using constant-time models.”
Distilling the Knowledge of BERT for Text Generation (2019, methodology): “Concurrent with our work, Ghazvininejad et al. (2019) proposed a similar conditional MLM for constant-time translation, and Yang et al. (2019) studied how to ﬁne-tune BERT for NMT.”
FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow (2019, methodology): “Knowledge Distillation Previous work on nonautoregressive generation (Gu et al., 2018; Ghazvininejad et al., 2019) has used translations produced by a pre-trained autoregressive NMT model as the training data, noting that this can significantly improve the performance.”