RobBERT: a Dutch RoBERTa-based Language Model

By Pieter Delobelle, Thomas Winters, Bettina Berendt. 2020

295 citations recorded by the source corpus.

Pre-trained language models have been dominating the field of natural language processing in recent years, and have led to significant performance gains for various complex natural language tasks. One of the most prominent pre-trained language models is BERT, which was released as an English as well as a multilingual version. Although multilingual BERT performs well on many tasks, recent studies show that BERT models trained on a single language significantly outperform the multilingual version. Training a Dutch BERT model thus has a lot of potential for a wide range of Dutch NLP tasks. While previous approaches have used earlier implementations of BERT to train a Dutch version of BERT, we used RoBERTa, a robustly optimized BERT approach, to train a Dutch language model called RobBERT. We measured its performance on various tasks as well as the importance of the fine-tuning dataset size. We also evaluated the importance of language-specific tokenizers and the model’s fairness. We found that RobBERT improves state-of-the-art results for various tasks, and especially significantly outperforms other models when dealing with smaller datasets. These results indicate that it is a powerful pre-trained model for a large variety of Dutch language tasks. The pre-trained and fine-tuned models are publicly available to support further downstream Dutch NLP applications.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, background): “…has been to release dozens of models that have instead been pre-trained on a single non-English language (Carmo et al., 2020; de Vries et al., 2019; Le et al., 2019; Martin et al., 2019; Delobelle et al., 2020; Malmsten et al., 2020; Nguyen and Nguyen, 2020; Polignano et al., 2019, etc.).”
Pre-trained models for natural language processing: A survey (2020, background): “Language-Specific BERT-wwm [26], ZEN [33], NEZHA [184] , ERNIE(Baidu) [157], BERTje [29], CamemBERT [112], FlauBERT [88], RobBERT [31]”
What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers (2021, background): “Although multilingual language models have been publicly available (Devlin et al., 2019), language-speciﬁc language models are still in demand, as they provide an edge over language-agnostic models (Martin et al., 2020; Nguyen and Nguyen, 2020; Delobelle et al., 2020).”
FlauBERT: Unsupervised Language Model Pre-training for French (2019, methodology): “…Recently, more monolingual BERT-based models have been released, such as for Arabic (Antoun et al., 2020), Dutch (de Vries et al., 2019; Delobelle et al., 2020), Finnish (Virtanen et al., 2019), Italian (Polignano et al., 2019), Portuguese (Souza et al., 2019), Russian (Kuratov and…”
PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data (2020, background): “Recent works demonstrate that monolingual pretrained models perform better on tasks in that same language than models pretrained on multilingual corpora [20, 22, 11, 3, 23, 2, 14, 1, 8, 12].”
A Primer on Pretrained Multilingual Language Models (2021, unknown): “, 2020), BERTje (Dutch) (Delobelle et al., 2020), FinBERT (Finnish) (Rönnqvist et al.”
Measuring Fairness with Biased Rulers: A Survey on Quantifying Biases in Pretrained Language Models (2021, methodology): “For Dutch, a Germanic language with a gender system, Delobelle et al. (2020); Ch´avez Mulsa and Spanakis (2020) evaluated RobBERT, a Dutch language model.”
Spanish Pre-trained BERT Model and Evaluation Data (2023, methodology): “…that usually got better performance than multilingual models as it is the case with CamemBERT (Martin et al., 2019) and FlauBERT (Le et al., 2019) for French, BERTje (de Vries et al., 2019) and Rob-BERT (Delobelle et al., 2020) for Dutch, FinBERT (Virtanen et al., 2019) for Finish, to name a few.”
Bisociative Pivoting as a Method for Moderating User Response to Text Similarity in Next-item News Recommendation (2026, unknown): “The document embedding captures the overall semantic content by concatenating article title and body text, then encoding with an appropriate pre-trained transformer model (in our implementation, where we model Dutch language news, RoBERTa-v2-dutch-base [5]).”