UNKs Everywhere: Adapting Multilingual Language Models to New Scripts

By Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, Sebastian Ruder. 2020

155 citations recorded by the source corpus.

Massively multilingual language models such as multilingual BERT offer state-of-the-art cross-lingual transfer performance on a range of NLP tasks. However, due to limited capacity and large differences in pretraining data sizes, there is a profound performance gap between resource-rich and resource-poor target languages. The ultimate challenge is dealing with under-resourced languages not covered at all by the models and written in scripts unseen during pretraining. In this work, we propose a series of novel data-efficient methods that enable quick and effective adaptation of pretrained multilingual models to such low-resource languages and unseen scripts. Relying on matrix factorization, our methods capitalize on the existing latent knowledge about multiple languages already available in the pretrained model’s embedding matrix. Furthermore, we show that learning of the new dedicated embedding matrix in the target language can be improved by leveraging a small number of vocabulary items (i.e., the so-called lexically overlapping tokens) shared between mBERT’s and target language vocabulary. Our adaptation techniques offer substantial performance gains for languages with unseen scripts. We also demonstrate that they can yield improvements for low-resource languages written in scripts covered by the pretrained model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Ethical and social risks of harm from Language Models (2021, background): “Technical workarounds raise new challenges Various solutions are being explored to increase LM performance in different languages, such as translating a prompt to English, generating predictions in English, then translating these predictions back into the original language of the prompt (Caswell et ”
AdapterDrop: On the Efficiency of Adapters in Transformers (2020, methodology): “We use a batch size of 32 and, if not otherwise noted, the default hyperparameters for adapter ﬁne-tuning as in (Pfeiffer et al., 2021a).”
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, methodology): “This makes it difﬁcult to effectively transfer knowledge across tasks with all the tasks being solved equally well (Pfeiffer et al., 2020a), thus considerably limiting the applicability of multi-task learning in many scenarios.”
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting (2022, unknown): “Language adapter refers to the bottleneck adapter with down-and up-projection feedforward layers (Houlsby et al., 2019; Pfeiffer et al., 2021a) that are inserted into each Transformer block.”
Lifting the Curse of Multilinguality by Pre-training Modular Transformers (2022, methodology): “4We find that the residual connection proposed by Pfeiffer et al. (2021a) results in training instabilities when trained together with the transformer weights.”
What to Pre-Train on? Efficient Intermediate Task Selection (2021, methodology): “We mostly adhere to the hyperparameter recommendations of the Transformers library and Pfeiffer et al. (2021a) for adapter training.”
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, methodology): “Furthermore, we show that using language adapters (and invertible adapters) in conjunction with task adapters, as proposed by Pfeiffer et al. (2020c), is overall slightly more effective than using task adapters only.”
MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer (2021, methodology): “In this work, we adopt the competitive and lightweight (so-called bottleneck) adapter variant of Pfeiffer et al. (2021a).”
Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold (2022, background): “…languages with productive morphological systems also are disadvantaged when shared ‘languageuniversal’ tokenizers are used in current large-scale multilingual language models (Ács, 2019; Rust et al., 2021) without any further vocabulary adaptation (Wang et al., 2020; Pfeiffer et al., 2021).”
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants (2023, unknown): “10 For the Indo-10 Aryan languages (i.e. Bengali, Hindi, Nepali, Urdu), we hypothesized cross-lingual transfer would be higher for these languages in the Latin variant since the tok-enization will be more suitable and there is opportunity for shared subwords (anchor points) (Conneau et al., 2020b).”
Writing System and Speaker Metadata for 2,800+ Language Varieties (2022, background): “Information about writing systems is crucial not only for product decisions like keyboard layouts, but also for modeling, as pre-trained models have been shown to be affected by a language’s script (Muller et al., 2021; Rust et al., 2021; Pfeiffer et al., 2021).”
Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment (2023, background): “MAD-X [75], [107] uses language-specific, task-specific, and invertible adapter to learn language-specific and task-specific transformations, as well as address vocabulary mismatches between multilingual and target languages in a modular manner, enabling the adaptation of pretrained multilingual…”
mGPT: Few-Shot Learners Go Multilingual (2022, methodology): “The design of the tokenization method may have a significant impact on learning efficient representations, model memorization, and downstream performance (Mielke et al., 2021; Nogueira et al., 2021; Pfeiffer et al., 2021; Rust et al., 2021).”
A Primer on Pretrained Multilingual Language Models (2021, unknown): “In the context of the above discussion on adapters, we would like to point the readers to AdapterHub.ml 4 which is a useful repository containing all recent adapter architectures (Pfeiffer et al., 2020a).”