FlauBERT: Unsupervised Language Model Pre-training for French

By Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexander Allauzen, Benoit Crabb'e, Laurent Besacier, Didier Schwab. 2019

441 citations recorded by the source corpus.

Language models have become a key step to achieve state-of-the art results in many different Natural Language Processing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way to pre-train continuous word representations that can be fine-tuned for a downstream task, along with their contextualization at the sentence level. This has been widely demonstrated for English using contextualized representations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019; Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large and heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for Scientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks (text classification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most of the time they outperform other pre-training approaches. Different versions of FlauBERT as well as a unified evaluation protocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared to the research community for further reproducible experiments in French NLP.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, background): “…has been to release dozens of models that have instead been pre-trained on a single non-English language (Carmo et al., 2020; de Vries et al., 2019; Le et al., 2019; Martin et al., 2019; Delobelle et al., 2020; Malmsten et al., 2020; Nguyen and Nguyen, 2020; Polignano et al., 2019, etc.).”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, positive): “To load FlauBERT (Le et al., 2020), a BERT model pretrained on a French training corpus, the command is:”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “To load FlauBERT (Le et al., 2020), a BERT model pretrained on a French training corpus, the command is:”
Pre-trained models for natural language processing: A survey (2020, neutral): “Although multilingual PTMs perform well on many languages, recent work showed that PTMs trained on a single language significantly outperform the multilingual results [125, 95, 186].”
CamemBERT: a Tasty French Language Model (2019, background): “(Le et al., 2019; Virtanen et al., 2019; Delobelle et al., 2020), for as much as 30 languages (Nozza et al.”
BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla (2021, methodology): “We trained a Wordpiece (Wu et al., 2016) vocabulary of 32k subword tokens on the resulting corpus with a 400 character alphabet, kept larger than the native Bangla alphabet to capture code-switching (Poplack, 1980) and allow romanized Bangla contents for better generalization.”
Transformers: State-of-the-Art Natural Language Processing (2020, unknown): “To load FlauBERT (Le et al., 2020), a BERT model pretrained on a French training corpus, the command is:”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “benchmarks are introduced to evaluate the progress in other monolingual [88], [92], [135], [136], [140], [147], [151], [152], [272]–[274] as well as multilingual models [112], [267], [270].”
BERTje: A Dutch BERT Model (2019, methodology): “Monolingual models with the BERT architecture have been developed for Italian (Polignano et al., 2019), French (Le et al., 2019), German, Finnish (Virtanen et al., 2019), and Japanese.”
Spanish Pre-trained BERT Model and Evaluation Data (2023, methodology): “This has led to the release of BERT models in Russian (Kuratov & Arkhipov, 2019), French (Martin et al., 2019; Le et al., 2019), Dutch (de Vries et al., 2019; Delobelle et al., 2020), Italian (Polignano et al., 2019), and Por-tugese (Souza et al., 2019).”
Language Models are Few-shot Multilingual Learners (2021, methodology): “…models with self-supervised learning using un-labelled monolingual text data collected from various resources available online (Wilie et al., 2020; Le et al., 2020; Martin et al., 2020; Eddine et al., 2020; Nguyen and Nguyen, 2020; Scheible et al., 2020; Bhattacharjee et al., 2021; Lee et al.,…”
ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic (2020, methodology): “(2020) and FlauBERT (Le et al., 2020) for French, PhoBERT for Vietnamese (Nguyen and Nguyen, 2020), and”
Civil Rephrases Of Toxic Texts With Self-Supervised Transformers (2021, background): “Besides, controlled text generation (Ficler and Goldberg, 2017; Keskar et al., 2019; Le et al., 2019; Dathathri et al., 2020) is a NLG task that consists of a language model conditioned on the attributes of the generated text such as the style.”
German’s Next Language Model (2020, background): “, 2020) and FlauBERT (Le et al., 2020) for French, Finnish BERT (Virtanen et al.”
IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding (2020, background): “FLUE (Le et al., 2019) is an evaluation NLP benchmark for the French language divided into six different task categories: text classiﬁcation, paraphrasing, NLI, parsing, POS tagging, and word sense disambiguation.”
iNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages (2020, methodology): “Benchmarks such as GLUE (Wang et al., 2018), SuperGLUE (Wang et al., 2019), CLUE (Chinese) (Xu et al., 2020), and FLUE (French) (Le et al., 2020) are important for tracking the efficacy of NLP models across languages.”