How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models

By Phillip Rust, Jonas Pfeiffer, Ivan Vulic, Sebastian Ruder, Iryna Gurevych. 2020

419 citations recorded by the source corpus.

In this work, we provide a systematic and comprehensive empirical comparison of pretrained multilingual language models versus their monolingual counterparts with regard to their monolingual task performance. We study a set of nine typologically diverse languages with readily available pretrained monolingual models on a set of five diverse monolingual downstream tasks. We first aim to establish, via fair and controlled comparisons, if a gap between the multilingual and the corresponding monolingual representation of that language exists, and subsequently investigate the reason for any performance difference. To disentangle conflating factors, we train new monolingual models on the same data, with monolingually and multilingually trained tokenizers. We find that while the pretraining data size is an important factor, a designated monolingual tokenizer plays an equally important role in the downstream performance. Our results show that languages that are adequately represented in the multilingual model’s vocabulary exhibit negligible performance decreases over their monolingual counterparts. We further find that replacing the original multilingual tokenizer with the specialized monolingual tokenizer improves the downstream performance of the multilingual model for almost every task and language.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “A very high fertility on a language compared to a monolingual tokenizer may indicate a degradation on the downstream multilingual performance of the model (Rust et al., 2021).”
UNKs Everywhere: Adapting Multilingual Language Models to New Scripts (2020, background): “These findings have been validated by Rust et al. (2020) for even higher resource languages.”
Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold (2022, background): “also are disadvantaged when shared ‘languageuniversal’ tokenizers are used in current large-scale multilingual language models (Ács, 2019; Rust et al., 2021) without any further vocabulary adaptation (Wang et al.”
A Primer on Pretrained Multilingual Language Models (2021, neutral): “Tokenization Apart from the pretraining corpora size, Rust et al. (2021) show that the monolingual models perform better than multilingual models due to their language speciﬁc tokenizer.”
MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer (2021, background): “(3)Other examples include the training of language-specific pretrained language models (Rust et al., 2021) as well as language pair-specific encoder–decoder models for machine translation (Luong et al.”
Lifting the Curse of Multilinguality by Pre-training Modular Transformers (2022, background): “…cross-lingual supervision (e.g., parallel data031 or translation dictionaries; Pires et al., 2019; Wu032 and Dredze, 2019; Artetxe et al., 2020; Hu et al.,033 2020; K et al., 2020; Rust et al., 2021).034 However, recent work has uncovered fundamen-035 tal limitations of multilingual transformers.”
When does pretraining help?: assessing self-supervised learning for law and the CaseHOLD dataset of 53,000+ legal holdings (2021, methodology): “Just as using an in-domain tokenizer helps in multilingual settings [39], using a custom tokenizer should improve performance consistently for the “language of law.”
The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models (2021, background): “Rust et al. (2020) empirically compared multilingual pre-trained language models to their monolingual counterparts on a set of nine typologically diverse languages.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “adequate representation in model vocabulary is on par with their monolingual models [134].”
XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond (2021, neutral): “gual models tend to underperform monolingual models in language-specific tasks (Rust et al., 2020).”
What to Pre-Train on? Efficient Intermediate Task Selection (2021, background): “2 Besides being more efficient, adapters are also highly modular, enabling a wider range of transfer learning techniques (Pfeiffer et al., 2020b, 2021a,b; Üstün et al., 2020; Vidoni et al., 2020; Rust et al., 2021; Ansell et al., 2021).”
Writing System and Speaker Metadata for 2,800+ Language Varieties (2022, background): “Information about writing systems is crucial not only for product decisions like keyboard layouts, but also for modeling, as pre-trained models have been shown to be affected by a language’s script (Muller et al., 2021; Rust et al., 2021; Pfeiffer et al., 2021).”
mGPT: Few-Shot Learners Go Multilingual (2022, methodology): “The design of the tokenization method may have a significant impact on learning efficient representations, model memorization, and downstream performance (Mielke et al., 2021; Nogueira et al., 2021; Pfeiffer et al., 2021; Rust et al., 2021).”
BERT2D: Two Dimensional Positional Embeddings for Efficient Turkish NLP (2024, methodology): “For fine-tuning, we used hyperparameters as close as possible to [29].”