Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language

By Yuri Kuratov, Mikhail Arkhipov. arXiv.org, 2019

326 citations recorded by the source corpus.

The paper introduces methods of adaptation of multilingual masked language models for a specific language. Pre-trained bidirectional language models show state-of-the-art performance on a wide range of tasks including reading comprehension, natural language inference, and sentiment analysis. At the moment there are two alternative approaches to train such models: monolingual and multilingual. While language specific models show superior performance, multilingual models allow to perform a transfer from one language to another and solve tasks for different languages simultaneously. This work shows that transfer learning from a multilingual model to monolingual model results in significant growth of performance on such tasks as reading comprehension, paraphrase detection, and sentiment analysis. Furthermore, multilingual initialization of monolingual model substantially reduces training time. Pre-trained models for the Russian language are open sourced.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/cc94d15ba408c260c8fe4fa4f1cb6797a996dd21

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Spanish Pre-trained BERT Model and Evaluation Data (2023, methodology): “This has led to the release of BERT models in Russian (Kuratov & Arkhipov, 2019), French (Martin et al., 2019; Le et al., 2019), Dutch (de Vries et al., 2019; Delobelle et al., 2020), Italian (Polignano et al., 2019), and Por-tugese (Souza et al., 2019).”
Pre-trained models for natural language processing: A survey (2020, background): “Kuratov and Arkhipov [90] used transfer learning techniques to adapt a multilingual PTM to a monolingual PTM for Russian language.”
FlauBERT: Unsupervised Language Model Pre-training for French (2019, background): “, 2019), Russian (Kuratov and Arkhipov, 2019), Spanish (Cañete et al.”
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models (2020, background): “…2019; Conneau et al., 2019), 5 and 8 monolingual pre-trained LMs, BERT (en), CamemBERT (fr), BERTje (nl), BETO (es), Ru-BERT (ru), Chinese BERT (zh), BERTurk (tr), and GreekBERT (el) (Martin et al., 2020; de Vries et al., 2019; Cañete et al., 2020; Kuratov and Arkhipov, 2019; Schweter, 2020).”
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, methodology): “, 2020) Russian RU Indo-European RuBERT (Kuratov and Arkhipov, 2019) Turkish TR Turkic BERTurk (Schweter, 2020)”
PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data (2020, background): “Recent works demonstrate that monolingual pretrained models perform better on tasks in that same language than models pretrained on multilingual corpora [20, 22, 11, 3, 23, 2, 14, 1, 8, 12].”