Should You Mask 15% in Masked Language Modeling?

By Alexander Wettig, Tianyu Gao, Zexuan Zhong, Danqi Chen. 2022

213 citations recorded by the source corpus.

Masked language models (MLMs) conventionally mask 15% of tokens due to the belief that more masking would leave insufficient context to learn good representations; this masking rate has been widely used, regardless of model sizes or masking strategies. In this work, we revisit this important choice of MLM pre-training. We first establish that 15% is not universally optimal, and larger models should adopt a higher masking rate. Specifically, we find that masking 40% outperforms 15% for BERT-large size models on GLUE and SQuAD. Interestingly, an extremely high masking rate of 80% can still preserve 95% fine-tuning performance and most of the accuracy in linguistic probing, challenging the conventional wisdom about the role of the masking rate. We then examine the interplay between masking rates and masking strategies and find that uniform masking requires a higher masking rate compared to sophisticated masking strategies such as span or PMI masking. Finally, we argue that increasing the masking rate has two distinct effects: it leads to more corruption, which makes the prediction task more difficult; it also enables more predictions, which benefits optimization. Using this framework, we revisit BERT’s 80-10-10 corruption strategy. Together, our results contribute to a better understanding of MLM pre-training.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

UL2: Unifying Language Learning Paradigms (2022, methodology): “This slightly conflicts with the finding of (Wettig et al., 2022) although our architectures use a inputs-totargets form of pretraining instead of BERT-style masked language modeling.”
Training Trajectories of Language Models Across Scales (2022, background): “Regarding the pre-training/fine-tuning paradigm, Wet- tig et al. (2022) and Tay et al. (2022) find that a lower pre-training perplexity does not necessarily translate to better fine-tuning performance.”
Prompt Injection: Parameterization of Fixed Inputs (2022, unknown): “We set the mask ratio as the difficulty criteria [34] and gradually increase the ratio throughout the Continued Pre-training.”
Unifying Language Learning Paradigms (2022, methodology): “Related to our proposed X-denoisers, (Wettig et al., 2022) studies the effect of corruption rate in BERT-style masked language modeling and hypothesizes that this improves sample efﬁciency along with beneﬁtting larger models.”
A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, background): “It is worth mentioning that this also motivates a recent work to experiment with a higher masking rate in masked language modeling for higher effectiveness [48].”
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2024, unknown): “We remove the Next-Sentence Prediction objective which introduces noticeable overhead for no performance improvement (Liu et al., 2019a; Izsak et al., 2021), and use a masking rate of 30 percent, as the original rate of 15 percent has since been shown to be sub-optimal (Wettig et al., 2023).”
NextLevelBERT: Masked Language Modeling with Higher-Level Representations for Long Documents (2024, background): “To verify if these findings translate to NextLevel-BERT, we compare BERT’s original masking strategy versus the recommendations by Wettig et al. (2023) with masking rates between 15-40%.”