BERTje: A Dutch BERT Model

By Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, Malvina Nissim. 2019

329 citations recorded by the source corpus.

The transformer-based pre-trained language model BERT has helped to improve state-of-the-art performance on many natural language processing (NLP) tasks. Using the same architecture and parameters, we developed and evaluated a monolingual Dutch BERT model called BERTje. Compared to the multilingual BERT model, which includes Dutch but is only based on Wikipedia text, BERTje is based on a large and diverse dataset of 2.4 billion tokens. BERTje consistently outperforms the equally-sized multilingual BERT model on downstream NLP tasks (part-of-speech tagging, named-entity recognition, semantic role labeling, and sentiment analysis). Our pre-trained Dutch BERT model is made available at this https URL.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, background): “…English-centricity has been to release dozens of models that have instead been pre-trained on a single non-English language (Carmo et al., 2020; de Vries et al., 2019; Le et al., 2019; Martin et al., 2019; Delobelle et al., 2020; Malmsten et al., 2020; Nguyen and Nguyen, 2020; Polignano et al.,…”
Pre-trained models for natural language processing: A survey (2020, background): “StructBERT [193] and BERTje [33] also take SOP as their self-supervised learning task.”
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models (2020, background): “…2019; Conneau et al., 2019), 5 and 8 monolingual pre-trained LMs, BERT (en), CamemBERT (fr), BERTje (nl), BETO (es), Ru-BERT (ru), Chinese BERT (zh), BERTurk (tr), and GreekBERT (el) (Martin et al., 2020; de Vries et al., 2019; Cañete et al., 2020; Kuratov and Arkhipov, 2019; Schweter, 2020).”
FlauBERT: Unsupervised Language Model Pre-training for French (2019, methodology): “…Recently, more monolingual BERT-based models have been released, such as for Arabic (Antoun et al., 2020), Dutch (de Vries et al., 2019; Delobelle et al., 2020), Finnish (Virtanen et al., 2019), Italian (Polignano et al., 2019), Portuguese (Souza et al., 2019),…”
RobBERT: a Dutch RoBERTa-based Language Model (2020, methodology): “We replicated the high-level sentiment analysis task used to evaluate BERTje (de Vries et al., 2019) to be able to compare our methods.”
PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data (2020, background): “Recent works demonstrate that monolingual pretrained models perform better on tasks in that same language than models pretrained on multilingual corpora [20, 22, 11, 3, 23, 2, 14, 1, 8, 12].”
ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic (2020, methodology): “…outperformed by mono-lingual models pre-trained with larger vocabulary and bigger language-speciﬁc datasets (Virtanen et al., 2019; Antoun et al., 2020; Dadas et al., 2020; † All authors contributed equally. de Vries et al., 2019; Le et al., 2020; Martin et al., 2020; Nguyen and Tuan Nguyen, 2020).”
AraBERT: Transformer-based Model for Arabic Language Understanding (2020, background): “…a multilingual model was provided for the public, pre-training monolingual BERT for other languages proved to provide better performance over the multilingual BERT such as Italian BERT Al-berto (Polignano et al., 2019) and other publicly available BERTs (Martin et al., 2019; de Vries et al., 2019).”
Overview of the Transformer-based Models for NLP Tasks (2020, background): “For example BERTje [36] is a Dutch version, AlBERTo [37] is an Italian version, and CamemBERT [38] and FlauBERT [39] are two different models for French.”
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, background): “Qualitatively and at first glance, our results displayed in Table 2 seem to confirm the prevailing view that monolingual models are more effective than multilingual models (Rönnqvist et al., 2019; Antoun et al., 2020; de Vries et al., 2019; Virtanen et al., 2019, inter alia).”
A Primer on Pretrained Multilingual Language Models (2021, unknown): “…leading to many language speciﬁc BERTs such as FlauBERT (French) (Le et al., 2020), CamemBERT (French) (Martin et al., 2020), BERTje (Dutch) (de Vries et al., 2019), FinBERT (Finnish) (Rönnqvist et al., 2019), BERTeus (Basque) (Agerri et al., 2020), AfriBERT (Afrikaans) (Ralethe, 2020),…”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “trained based on standard model architectures like GPT [142] BERT [74], [75], [92], [133], [135], [136], [139]–[141], [144]–[146], [151]–[154], RoBERTa [137], [138], [147]– [149], [152], [155], ALBERT [92], [150], ELECTRA [132], [143], and T5 [77].”
PhoBERT: Pre-trained language models for Vietnamese (2020, methodology): “For other languages, one could retrain a language-speciﬁc model using the BERT architecture [ Vu et al. , 2019; Martin et al. , 2019; de Vries et al. , 2019 ] or employ existing pre-trained multi-lingual BERT-based models [ Devlin et al. , 2019; Conneau et al. , 2019; Conneau and Lample, 2019 ] .”