Playing with Words at the National Library of Sweden - Making a Swedish BERT

By Martin Malmsten, Love Börjeson, Chris Haffenden. 2020

141 citations recorded by the source corpus.

This paper introduces the Swedish BERT ("KB-BERT") developed by the KBLab for data-driven research at the National Library of Sweden (KB). Building on recent efforts to create transformer-based BERT models for languages other than English, we explain how we used KB's collections to create and train a new language-specific BERT model for Swedish. We also present the results of our model in comparison with existing models - chiefly that produced by the Swedish Public Employment Service, Arbetsformedlingen, and Google's multilingual M-BERT - where we demonstrate that KB-BERT outperforms these in a range of NLP tasks from named entity recognition (NER) to part-of-speech tagging (POS). Our discussion highlights the difficulties that continue to exist given the lack of training data and testbeds for smaller languages like Swedish. We release our model for further exploration and research here: this https URL .

Subfields: Artificial Intelligence, Literature and Literary Theory, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Colin Raffel
2. Noah Constant
3. Rami Al-Rfou
4. Adam Roberts
5. Mohit Bansal
6. Mihir Kale
7. Percy Liang
8. Luke S. Zettlemoyer

In Literature and Literary Theory

1. Muhammad Abdul-Mageed
2. AbdelRahim Elmadany
3. El Moatez Billah Nagoudi

In Large Language Models

1. Colin Raffel
2. Mihir Kale
3. Noah Constant
4. Rami Al-Rfou
5. Adam Roberts
6. Aditya Siddhant
7. Linting Xue
8. Muhammad Abdul-Mageed

In Natural Language Processing

1. Colin Raffel
2. Noah Constant
3. Rami Al-Rfou
4. Mihir Kale
5. Aditya Siddhant
6. Adam Roberts
7. Linting Xue
8. Aditya Barua

Citations in the corpus

• mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, background): “…has been to release dozens of models that have instead been pre-trained on a single non-English language (Carmo et al., 2020; de Vries et al., 2019; Le et al., 2019; Martin et al., 2019; Delobelle et al., 2020; Malmsten et al., 2020; Nguyen and Nguyen, 2020; Polignano et al., 2019, etc.).”
• ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic (2020, methodology): “…outperformed by mono-lingual models pre-trained with larger vocabulary and bigger language-specific datasets (Virtanen et al., 2019; Antoun et al., 2020; Dadas et al., 2020; † All authors contributed equally. de Vries et al., 2019; Le et al., 2020; Martin et al., 2020; Nguyen and Tuan Nguyen, 2020).”
• Cross-lingual and Multilingual CLIP (2022, methodology): “Additionally, we train two versions of a Swedish-only encoder by starting from the Swedish KB-BERT (Malmsten et al., 2020).”