Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little

By Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, Douwe Kiela. 2021

297 citations recorded by the source corpus.

A possible explanation for the impressive performance of masked language model (MLM) pre-training is that such models have learned to represent the syntactic structures prevalent in classical NLP pipelines. In this paper, we propose a different explanation: MLMs succeed on downstream tasks almost entirely due to their ability to model higher-order word co-occurrence statistics. To demonstrate this, we pre-train MLMs on sentences with randomly shuffled word order, and show that these models still achieve high accuracy after fine-tuning on many downstream tasks—including tasks specifically designed to be challenging for models that ignore word order. Our models perform surprisingly well according to some parametric syntactic probes, indicating possible deficiencies in how we test representations for syntactic information. Overall, our results show that purely distributional information largely explains the success of pre-training, and underscore the importance of curating challenging evaluation datasets that require deeper linguistic knowledge.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Natural Language Processing

Citations in the corpus

True Few-Shot Learning with Language Models (2021, methodology): “Recent work has used MDL to determine which learning algorithms are most effective at explaining the given data [Rissanen Data Analysis; 10, 34].”
Transformer Language Models without Positional Encodings Still Learn Positional Information (2022, neutral): “This result echoes the ﬁndings of Sinha et al. (2021), who also observed that MLMs without positional embeddings suffer signiﬁcant performance degradation.”
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, background): “While Sinha et al. (2021) found that natural word order during pre-training matters little for downstream performance on tasks in benchmarks like GLUE (Wang et al., 2018), they also found that randomizing word order greatly reduced model preferences for correct inflections in syntactic evaluation…”
Impact of Pretraining Term Frequencies on Few-Shot Reasoning (2022, background): “Sinha et al. (2021) further demonstrate that shufﬂing word order during pretraining has minimal impact on an LMs’ accuracy on downstream tasks, and, concurrent with this work, Min et al. (2022) similarly ﬁnd that shufﬂing labels in in-context learning demonstrations has a minimal impact on few-shot…”
The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models (2021, methodology): “This agrees with experiments shuffling the order of words (Pham et al., 2020; Sinha et al., 2021) finding that modern language models tend to focus mainly on higherorder word co-occurrences, rather than word order, and suggests that word-order information is underutilized in state-of-the-art languag”
Sometimes We Want Ungrammatical Translations (2021, unknown): “(Si et al., 2019; Sinha et al., 2020; Pham et al., 2020; Gupta et al., 2021; Sinha et al., 2021).”
Emergent Structures and Training Dynamics in Large Language Models (2022, background): “Sinha et al. (2021) use a variety of word order permu-441 tations that preserve distributional information to isolate whether what language models learn is ac-443 tually syntax.”
UnNatural Language Inference (2020, background): “Most relatedly, several concurrent works (Pham et al., 2020; Alleman et al., 2021; Gupta et al., 2021; Sinha et al., 2021; Parthasarathi et al., 2021) investigated the effect of word order permutations on transformer NNs.”
When and why vision-language models behave like bags-of-words, and what to do about it? (2022, background): “Order information in language and vision: Several works have highlighted the lack of word order sensitivity in large language models (Hessel & Schoﬁeld, 2021; O’Connor & Andreas, 2021; Pham et al., 2021; Sinha et al., 2021).”
A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervision (2021, background): “While [41] shows that even large-scale pre-training is invariant to word order shufﬂe, we did not have the resources to verify it in our setting.”
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning (2021, unknown): “Other papers ablate various aspects of pretraining, such as changing the masking scheme [14, 20, 40] or permuting the word order [32].”