CamemBERT: a Tasty French Language Model

By Louis Martin, Benjamin Muller, Pedro Ortiz Suarez, Yoann Dupont, Laurent Romary, Eric Villemonte de la Clergerie, Djamé Seddah, Benoît Sagot. 2019

1,110 citations recorded by the source corpus.

Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available models have either been trained on English data or on the concatenation of data in multiple languages. This makes practical use of such models –in all languages except English– very limited. In this paper, we investigate the feasibility of training monolingual Transformer-based language models for other languages, taking French as an example and evaluating our language models on part-of-speech tagging, dependency parsing, named entity recognition and natural language inference tasks. We show that the use of web crawled data is preferable to the use of Wikipedia data. More surprisingly, we show that a relatively small web crawled dataset (4GB) leads to results that are as good as those obtained using larger datasets (130+GB). Our best performing model CamemBERT reaches or improves the state of the art in all four downstream tasks.

Subfields: Artificial Intelligence, Information Systems, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Large Language Models

In Natural Language Processing

Citations in the corpus

mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, background): “…has been to release dozens of models that have instead been pre-trained on a single non-English language (Carmo et al., 2020; de Vries et al., 2019; Le et al., 2019; Martin et al., 2019; Delobelle et al., 2020; Malmsten et al., 2020; Nguyen and Nguyen, 2020; Polignano et al., 2019, etc.).”
Big Bird: Transformers for Longer Sequences (2020, methodology): “…based tasks – as a seq2seq model for translation [92], summarization [66], generation [15], etc. or as a standalone encoders for sentiment analysis [84], POS tagging [65], machine reading comprehension [94], etc. – and it is known to vastly outperform previous sequence models like LSTM [37].”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “The monolingual tokenizers we used were the ones from CamemBERT (Martin et al., 2020), GPT-2 (Radford et al.”
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, methodology): “OSCAR has also been used in independent studies to train monolingual or multilingual language models (ar, as, bn, de, el, fr, gu, he, hi, kn, ml, mr, nl, or, pa, ro, ta, te) and subsequently evaluate them on various downstream tasks (Antoun et al., 2021; Kakwani et al., 2020; Wilie et al., 2020; Cha”
Pre-trained models for natural language processing: A survey (2020, background): “Although multilingual PTMs perform well on many languages, recent work showed that PTMs trained on a single language significantly outperform the multilingual results [125, 95, 186].”
What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers (2021, background): “Although multilingual language models have been publicly available (Devlin et al., 2019), language-speciﬁc language models are still in demand, as they provide an edge over language-agnostic models (Martin et al., 2020; Nguyen and Nguyen, 2020; Delobelle et al., 2020).”
When Do You Need Billions of Words of Pretraining Data? (2020, background): “Martin et al. (2020) also show that French MLMs can already learn a lot from small-scale pretraining.”
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages (2020, methodology): “, 2019) as well as a French BERT model (called CamemBERT) (Martin et al., 2019) have also been released.”
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models (2020, background): “…2019; Conneau et al., 2019), 5 and 8 monolingual pre-trained LMs, BERT (en), CamemBERT (fr), BERTje (nl), BETO (es), Ru-BERT (ru), Chinese BERT (zh), BERTurk (tr), and GreekBERT (el) (Martin et al., 2020; de Vries et al., 2019; Cañete et al., 2020; Kuratov and Arkhipov, 2019; Schweter, 2020).”
FlauBERT: Unsupervised Language Model Pre-training for French (2019, methodology): “We compare the performance of FlauBERT with Multilingual BERT (Devlin et al., 2019, mBERT) and CamemBERT (Martin et al., 2019) on all tasks.”
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023, background): “, for Spanish (MMG, 2021), Polish (Resources and Technology Infrastructure, 2021), French (Martin et al., 2020; Kamal Eddine et al., 2021), Japanese (Wongso, 2021), Hindi (Parmar, 2021), and Sweddish (Moell, 2021).”
RobBERT: a Dutch RoBERTa-based Language Model (2020, methodology): “monolingual French CamemBERT model (Martin et al., 2019) also outperformed mBERT on all tasks.”
PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data (2020, background): “Recent works demonstrate that monolingual pretrained models perform better on tasks in that same language than models pretrained on multilingual corpora [20, 22, 11, 3, 23, 2, 14, 1, 8, 12].”
BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla (2021, methodology): “Motivated by the triumph of language-speciﬁc models (Martin et al. (2020); Polignano et al. (2019); Canete et al. (2020); An-toun et al. (2020), inter alia ) over multilingual models in many other languages, in this work, we present BanglaBERT – a BERT-based (Devlin et al., 2019) Bangla NLU model…”
Domain-specific Genetic Algorithm for Multi-tenant DNNAccelerator Scheduling (2021, neutral): “Lang ALBERT [59], Bart [62], BERT [24], CamemBERT [68], CTRL [52], DistilBERT [91], ELECTRA [18], FlauBERT [60], GPT2 [82], GPT [81], LongFormer [6], MarianNMT [49], MobileBERT [103], Reformer [53], RetriBERT [44], RoBERTa [64], T5 [83], TransformerXL [22], XLM [58], XLM-RoBERTa [19], XLNet [119] Re”
Spanish Pre-trained BERT Model and Evaluation Data (2023, methodology): “Several single-language BERT models came with results that usually got better performance than multilingual models as it is the case with CamemBERT (Martin et al., 2019) and FlauBERT (Le et al., 2019) for French, BERTje (de Vries et al., 2019) and Rob-BERT (Delobelle et al., 2020) for Dutch,…”
Towards a Cleaner Document-Oriented Multilingual Crawled Corpus (2022, methodology): “co (Martin et al., 2020) a contextualized model for French successfully used only Crawled data for pre-training, and even large generative language models like T5 have used mainly crawled data successfully (Raffel et al.”
Overview of the Transformer-based Models for NLP Tasks (2020, background): “For example BERTje [36] is a Dutch version, AlBERTo [37] is an Italian version, and CamemBERT [38] and FlauBERT [39] are two different models for French.”
Multilingual is not enough: BERT for Finnish (2019, background): “1 Sub-sequently, monolingual BERT models have been published for German 2 and French (Martin et al., 2019).”