BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling

By Javier de la Rosa, E. G. Ponferrada, Paulo Villegas, Pablo González de Prado Salas, Manu Romero, María Grandury. 2022

113 citations recorded by the source corpus.

The pre-training of large language models usually requires massive amounts of resources, both in terms of computation and data. Frequently used web sources such as Common Crawl might contain enough noise to make this pre-training sub-optimal. In this work, we experiment with different sampling methods from the Spanish version of mC4, and present a novel data-centric technique which we name $\textit{perplexity sampling}$ that enables the pre-training of language models in roughly half the amount of steps and using one fifth of the data. The resulting models are comparable to the current state-of-the-art, and even achieve better results for certain tasks. Our work is proof of the versatility of Transformers, and paves the way for small teams to train their models on a limited budget. Our models are available at this $\href{https://huggingface.co/bertin-project}{URL}$.

Subfields: Artificial Intelligence, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Efficiency & Compression

Citations in the corpus

Measuring Data (2022, methodology): “It was subsequently leveraged to evaluate the performance of language models (Brown et al., 1992; Bengio et al., 2000; Melis et al., 2017) as well as datasets, for tasks such as perplexity sampling (De la Rosa et al., 2022) and data selection (Toral et al., 2015).”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, methodology): “As in De la Rosa et al. (2022), we discarded documents to move the perplexity distribution towards the median, to avoid too high perplexity scores (deemed as not useful for the model), but subsampling was done by perplexity thresholding, not by reshaping the distribution as in De la Rosa et al. (202”