Masked Language Model Scoring

By Julian Salazar, Davis Liang, Toan Q. Nguyen, Katrin Kirchhoff. 2019

615 citations recorded by the source corpus.

Pretrained masked language models (MLMs) require finetuning for most NLP tasks. Instead, we evaluate MLMs out of the box via their pseudo-log-likelihood scores (PLLs), which are computed by masking tokens one by one. We show that PLLs outperform scores from autoregressive language models like GPT-2 in a variety of tasks. By rescoring ASR and NMT hypotheses, RoBERTa reduces an end-to-end LibriSpeech model’s WER by 30% relative and adds up to +1.7 BLEU on state-of-the-art baselines for low-resource translation pairs, with further gains from domain adaptation. We attribute this success to PLL’s unsupervised expression of linguistic acceptability without a left-to-right bias, greatly improving on scores from GPT-2 (+10 points on island effects, NPI licensing in BLiMP). One can finetune MLMs to give scores without masking, enabling computation in a single inference pass. In all, PLLs and their associated pseudo-perplexities (PPPLs) enable plug-and-play use of the growing number of pretrained MLMs; e.g., we use a single cross-lingual model to rescore translations in multiple languages. We release our library for language model scoring at https://github.com/awslabs/mlm-scoring.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models (2020, methodology): “To approximate p(U |M, θ), we adapt pseudolog-likehood MLM scoring (Wang and Cho, 2019; Salazar et al., 2020).”
An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models (2023, methodology): “Therefore, we leverage pseudo-perplexity proposed by Salazar et al. (2020) in place of perplexity for auto-encoder PTLMs.”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, methodology): “Using RoBERTa-large, we calculate the pseudolog-likelihood (Wang and Cho, 2019; Salazar et al., 2020; Nangia et al., 2020) of descriptor/noun phrases (i.e. “tall guy” in the sentence “This is a tall guy.”) on a sample of 500,000 sentences in which descriptors are randomly drawn and inserted into…”
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, methodology): “Our modiﬁed version of P ET uses masked language models (Devlin et al., 2019) to assign probabilities to sequences of text; this is similar to using them in a generative fashion (Wang and Cho, 2019) and has previously been investigated by Salazar et al. (2020) and Ghazvininejad et al. (2019).”
Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP (2021, methodology): “Nangia et al. (2020) use pseudo-log-likelihood (Wang and Cho, 2019; Salazar et al., 2020) to assign scores to sentences using MLMs. Bias in an MLM is then measured as the proportion of entries for which the MLM assigns a higher score to the more stereotypical sentence; an ideal model that does not…”
When Do You Need Billions of Words of Pretraining Data? (2020, methodology): “(2020a) and RoBERTaLARGE performance reported by Salazar et al. (2020).”
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models (2020, background): “…which is the sum be easily adapted to non-consecutive cases. of log probabilities of each predicted token conditioned on the other tokens (Salazar et al., 2020): v ( j − i + 1) = = i log c k , where c k is the con-ﬁdence (probability) of the k -th predicted token, and v ( m ) is the…”
Are Discrete Units Necessary for Spoken Language Modeling? (2022, methodology): “It is derived from the pseudo-loglikelihood score (PLL) for MLMs [21], which was shown to be an effective sentence scoring method for BERT models in many scenarios [22].”
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data? (2021, background): “In general, multi-word targets are difﬁcult to assess as PPL is not well-deﬁned for masked language models like BERT (Jiang et al., 2020a; Salazar et al., 2020).”
An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models (2021, methodology): “Nangia et al. (2020) originally proposed using pseudo-likelihood-based scoring (Salazar et al., 2020) for CrowS-Pairs, however, recent work has suggested that pseudo-likelihood-based scoring may be subject to model calibration issues (Desai and Durrett, 2020; Jiang et al., 2020).”
Detect and Perturb: Neutral Rewriting of Biased and Sensitive Text via Gradient-based Decoding (2021, positive): “Fluency: We use the Pseudo Log-Likelihood (PLL) of Salazar et al. (2020) to measure the ﬂu-ency of our generated model.”
Should You Mask 15% in Masked Language Modeling? (2022, methodology): “, 2020) using pseudo loglikelihood scoring (Salazar et al., 2020).”
Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset (2022, methodology): “We then compare whether each model is more likely to guess “Jane Doe” using MLM Score [103].”
Bias and Fairness in Large Language Models: A Survey (2023, methodology): “Several techniques leverage pseudo-log-likelihood (PLL) (Salazar et al., 2020; Wang & Cho, 2019) to score the probability of generating a token given other words in the sentence.”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, neutral): “[84] Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff.”
A Survey on Gender Bias in Natural Language Processing (2021, methodology): “Then, using pseudo-log-likelihood masked language model scoring [Salazar et al. 2020], they estimate the probability of the unmodified tokens conditioned on the ones.”