Mask-Predict: Parallel Decoding of Conditional Masked Language Models

By Marjan Ghazvininejad, Omer Levy, Yinhan Liu, Luke S. Zettlemoyer. 2019

724 citations recorded by the source corpus.

Most machine translation systems generate text autoregressively from left to right. We, instead, use a masked language modeling objective to train a model to predict any subset of the target words, conditioned on both the input text and a partially masked target translation. This approach allows for efficient iterative decoding, where we first predict all of the target words non-autoregressively, and then repeatedly mask out and regenerate the subset of words that the model is least confident about. By applying this strategy for a constant number of iterations, our model improves state-of-the-art performance levels for non-autoregressive and parallel decoding translation models by over 4 BLEU on average. It is also able to reach within about 1 BLEU point of a typical left-to-right transformer model, while decoding significantly faster.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

End-to-End Object Detection with Transformers (2020, background): “s.1. Compared to most previous work on direct set prediction, the main features of DETR are the conjunction of the bipartite matching loss and transformers with (non-autoregressive) parallel decoding [29,12,10,8]. In contrast, previous work focused on autoregressive decoding with RNNs [43,41,30,36,4”
Defending Against Neural Fake News (2019, background): “This also suggests that recent progress on generating text in any order (Gu et al., 2019; Stern et al., 2019; Ghazvininejad et al., 2019) may lead to models that evade a Grover discriminator.”
InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “Finally, our model lays a foundation for future work on supervised infilling & editing via model fine-tuning, as well as performing iterative decoding, where the model can be used to refine its own output [27].”
Simple and Effective Retrieve-Edit-Rerank Text Generation (2020, methodology): “We train the model to predict the similarity between y′ and y as the output of the [RANK] token, akin to predicting a token from a different vocabulary (Ghazvininejad et al., 2019).”
How Can We Know What Language Models Know? (2019, background): “…in particular, a broad suite of NLP tasks are used to analyze both syntactic and semantic properties, providing evidence that contextualized representations encode linguistic knowledge in different layers (Hewitt and Manning, 2019; Tenney et al., 2019a,b; Jawahar et al., 2019; Goldberg, 2019).”
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, methodology): “Our modiﬁed version of P ET uses masked language models (Devlin et al., 2019) to assign probabilities to sequences of text; this is similar to using them in a generative fashion (Wang and Cho, 2019) and has previously been investigated by Salazar et al. (2020) and Ghazvininejad et al. (2019).”
Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, methodology): “, 2019), Mask-Predict (Ghazvininejad et al., 2019), Levenstein Transformer (Gu et al.”
WaveGrad: Estimating Gradients for Waveform Generation (2020, methodology): “Edit-based models can also iteratively refine the outputs during inference (Lee et al., 2018; Ghazvininejad et al., 2019; Chan et al., 2020), while they do not rely on a (continuous) gradient-based sampler, they rely on a (discrete) edit-based sampler.”
Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding (2021, methodology): “In contrast, our proposed shallow aggressive decoding tries to accelerate the model inference through parallel autoregressive decoding which is related to some previous work (Ghazvininejad et al., 2019; Stern et al., 2018) in neural machine translation (NMT), and the imbalanced encoder-decoder archi”
MaskGIT: Masked Generative Image Transformer (2022, methodology): “Our work is inspired by bi-directional machine translation [16, 19, 20] in NLP, and our novelty lies in the proposed new masking strategy and decoding algorithm which, as substantiated by our experiments, are essential for image generation.”
SoundStorm: Efficient Parallel Audio Generation (2023, neutral): “…al., 2020; Choromanski et al., 2021; Xiong et al., 2021; Hawthorne et al., 2022), ii) non-autoregressive, parallel decoding schemes (Gu et al., 2017; Ghazvininejad et al., 2019; Chang et al., 2022), iii) custom architectures adapted to the special structure of the tokens produced by neural audio…”
E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT (2019, methodology): “We found it useful to iteratively reﬁne predictions during inference, similar to techniques from non-autoregressive Machine Translation (Ghazvininejad et al., 2019).”
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models (2020, methodology): “In addition, the conﬁdence value c in previous methods contains probabilities when the predictions are ﬁrst generated, which will become stale once the surrounding tokens change (Ghazvininejad et al., 2019).”
Structured Denoising Diffusion Models in Discrete State-Spaces (2021, background): “(Generative) Masked Language-Models (MLMs) are diffusion models: Generative Masked Language Models ([13], [47]) are generative models that generate text from a sequence of [MASK] tokens.”
X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers (2020, background): “– Mask-predict-K (Ghazvininejad et al., 2019): This requires K sampling steps.”
Blank Language Models (2020, methodology): “To utilize them in rewriting tasks would require one to specify the insertion length in advance and heuristically determine a generation order among masks (Ghazvininejad et al., 2019; Wu et al., 2019).”
Understanding Knowledge Distillation in Non-autoregressive Machine Translation (2019, methodology): “…a fair comparison of the actual ability of each NAT-based model, we test all the models based on greedy decoding without any advanced search algorithms (e.g. length beam (Ghazvininejad et al., 2019), noisy parallel decoding (Ma et al., 2019), or re-ranking from the teacher model (Gu et al., 2018)).”
EDITOR: An Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints (2020, methodology): “Edit operations such as substitution (Ghazvininejad et al., 2019) and insertion-deletion (Gu et al., 2019) have reduced the quality gap between non-autoregressive and autoregressive models.”
Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation (2020, methodology): “Distillation Following previous work on non-autoregressive translation (e.g., Ghazvininejad et al., 2019; Kasai et al., 2020; Saharia et al., 2020), we apply sequence-level knowledge distillation (Kim and Rush, 2016) by training every model in all directions on translations produced by a standard…”