Multilingual is not enough: BERT for Finnish

By Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, Sampo Pyysalo. arXiv.org, 2019

318 citations recorded by the source corpus.

Deep learning-based language models pretrained on large unannotated text corpora have been demonstrated to allow efficient transfer learning for natural language processing, with recent approaches such as the transformer-based BERT model advancing the state of the art across a variety of tasks. While most work on these models has focused on high-resource languages, in particular English, a number of recent efforts have introduced multilingual models that can be fine-tuned to address tasks in a large number of different languages. However, we still lack a thorough understanding of the capabilities of these models, in particular for lower-resourced languages. In this paper, we focus on Finnish and thoroughly evaluate the multilingual BERT model on a range of tasks, comparing it with a new Finnish BERT model trained from scratch. The new language-specific model is shown to systematically and clearly outperform the multilingual. While the multilingual model largely fails to reach the performance of previously proposed methods, the custom Finnish BERT model establishes new state-of-the-art results on all corpora for all reference tasks: part-of-speech tagging, named entity recognition, and dependency parsing. We release the model and all related resources created for this study with open licenses at this https URL .

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/477d66dcd2c08243dcc69822d6da7ec06393773a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

German’s Next Language Model (2020, background): “These include but are not limited to monolingual models such as CamemBERT (Martin et al., 2020) and FlauBERT (Le et al., 2020) for French, Finnish BERT (Virtanen et al., 2019) and German BERTs by DBMDZ2 and deepset3.”
ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic (2020, background): “…Dutch (de Vries et al., 2019), CamemBERT (Martin et al., 2020) and FlauBERT (Le et al., 2020) for French, PhoBERT for Vietnamese (Nguyen and Tuan Nguyen, 2020), and the models presented by Virtanen et al. (2019) for Finnish, Dadas et al. (2020) for Polish, and Malmsten et al. (2020) for Swedish.”
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, methodology): “For instance, there are monolingual BERT variants for Arabic (Antoun et al., 2020), French (Martin et al., 2020), Finnish (Virtanen et al., 2019), Dutch (de Vries et al., 2019), Italian (Polignano et al., 2019), to name only a few.”
MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer (2020, background): “…issue even for high-resource languages where state-of-the-art multilingual models underperform their monolingual variants (Eisenschlos et al., 2019; Virtanen et al., 2019; Nozza et al., 2020), and performance decreases further when moving down the list of languages towards lower-resource languages…”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “FinnishBERT [145] Finnish Scratch MLM, NSP Finnish News, Online discussion and Common Crawl Corpus SentencePiece (50K)”
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings (2022, background): “Even when researchers seek to include other languages, these purportedly multilingual models often underserve certain languages and communities (Kerri-555 son et al., 2018; Virtanen et al., 2019; Kreutzer et al., 2022; Gururangan et al., 2022).”
FlauBERT: Unsupervised Language Model Pre-training for French (2019, methodology): “…BERT-based models have been released, such as for Arabic (Antoun et al., 2020), Dutch (de Vries et al., 2019; Delobelle et al., 2020), Finnish (Virtanen et al., 2019), Italian (Polignano et al., 2019), Portuguese (Souza et al., 2019), Russian (Kuratov and Arkhipov, 2019), Spanish (Cañete et…”
A Primer on Pretrained Multilingual Language Models (2021, unknown): “…2020a; Wu and Dredze, 2020; Agerri et al., 2020; Virtanen et al., 2019; Rönnqvist et al., 2019; Ro et al., 2020; de Vargas Feijó and Moreira, 2020; Virtanen et al., 2019; Wang et al., 2020a; Wu and Dredze, 2020) which show that the right choice depends on various factors such as model capacity,…”
Scaling Data-Constrained Language Models (2023, background): “However, both for code and natural language models, mixing different (programming) languages has been reported to under-perform monolingual models [80, 113].”
Pre-trained models for natural language processing: A survey (2020, background): “Although multilingual PTMs perform well on many languages, recent work showed that PTMs trained on a single language significantly outperform the multilingual results [125, 95, 186].”
Playing with Words at the National Library of Sweden - Making a Swedish BERT (2020, methodology): “Following the lead of both the German (Branden et al., 2019) and Finnish BERT (Virtanen et al., 2019) projects, we chose a dictionary size of approximately 50,000 tokens, with the rationale that Swedish contains many compound words.”
Spanish Pre-trained BERT Model and Evaluation Data (2023, methodology): “…that usually got better performance than multilingual models as it is the case with CamemBERT (Martin et al., 2019) and FlauBERT (Le et al., 2019) for French, BERTje (de Vries et al., 2019) and Rob-BERT (Delobelle et al., 2020) for Dutch, FinBERT (Virtanen et al., 2019) for Finish, to name a few.”
BERTje: A Dutch BERT Model (2019, methodology): “Monolingual models with the BERT architecture have been developed for Italian (Polignano et al., 2019), French (Le et al., 2019), German, Finnish (Virtanen et al., 2019), and Japanese.”
PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data (2020, background): “Recent works demonstrate that monolingual pretrained models perform better on tasks in that same language than models pretrained on multilingual corpora [20, 22, 11, 3, 23, 2, 14, 1, 8, 12].”