The MultiBERTs: BERT Reproductions for Robustness Analysis

By Thibault Sellam, Steve Yadlowsky, Jason Wei, Naomi Saphra, A. D'Amour, Tal Linzen, Jasmijn Bastings, Iulia Turc, Jacob Eisenstein, Dipanjan Das and 2 more. 2021

110 citations recorded by the source corpus.

Experiments with pre-trained models such as BERT are often based on a single checkpoint. While the conclusions drawn apply to the artifact tested in the experiment (i.e., the particular instance of the model), it is not always clear whether they hold for the more general procedure which includes the architecture, training data, initialization scheme, and loss function. Recent work has shown that repeating the pre-training process can lead to substantially different performance, suggesting that an alternate strategy is needed to make principled statements about procedures. To enable researchers to draw more robust conclusions, we introduce the MultiBERTs, a set of 25 BERT-Base checkpoints, trained with similar hyper-parameters as the original BERT model but differing in random weight initialization and shuffling of training data. We also define the Multi-Bootstrap, a non-parametric bootstrap method for statistical inference designed for settings where there are multiple pre-trained models and limited test data. To illustrate our approach, we present a case study of gender bias in coreference resolution, in which the Multi-Bootstrap lets us measure effects that may not be detected with a single checkpoint. We release our models and statistical library along with an additional set of 140 intermediate checkpoints captured during pre-training to facilitate research on learning dynamics.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

On The Impact of Machine Learning Randomness on Group Fairness (2023, methodology): “These findings challenge the effectiveness of many bias mitigation algorithms [3, 48], and even the legitimacy of several fairness trends present in literature [51].”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “There are several model suites designed to answer this question, including the causal decoder Mistral suite (Karamcheti et al., 2021) and the multiBERT suite (Sellam et al., 2021).”
Datamodels: Predicting Predictions from Training Data (2022, background): “For example, Sellam et al. (2021) make this point explicitly in the context of BERT (Devlin et al., 2019) pre-trained language models.”
Analyzing Transformers in Embedding Space (2022, methodology): “To demonstrate this, we use MultiBERT [Sellam et al., 2022], which contains 25 different instantiations of BERT initialized from different random seeds.”
Measuring Causal Effects of Data Statistics on Language Model's 'Factual' Predictions (2022, methodology): “The average ATE across the MultiBERTs are 9.12, 3.73 and 17.83 for the three hypotheses, respectively, with small standard deviations (0.58-1.18).”
Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, background): “[94] Thibault Sellam, Steve Yadlowsky, Jason Wei, Naomi Saphra, Alexander D’Amour, Tal Linzen, Jasmijn Bastings, Iulia Turc, Jacob Eisenstein, Dipanjan Das, et al.”
Is neural language acquisition similar to natural? A chronological probing study (2022, methodology): “• We test two models (MultiBERT(Sellam et al., 2021) and T5(Raffel et al., 2019)) on existing 12 different probing tasks in morphology, syntax, and discourse and present an analysis of the models’ gradual learning of language phenomena, in comparison with the well-known facts about the acquisition…”
Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators (2022, unknown): “We would like to refer to a recent study (Sellam et al., 2022) which reveals that these small GLUE tasks are very unstable, and the same model pretrained with different random initialization can have 2− 5 points difference in performance upon fine-tuning on these small tasks, while in comparison,…”
All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality (2021, background): “With the recent release of the MultiBERT checkpoints (Sellam et al., 2021), future work can uncover whether rogue dimensions are a coinci-dental property of some models, or whether they are a requisite for good performance.”