PhoBERT: Pre-trained language models for Vietnamese

By Dat Quoc Nguyen, Anh Gia-Tuan Nguyen. 2020

469 citations recorded by the source corpus.

We present PhoBERT with two versions, PhoBERT-base and PhoBERT-large, the first public large-scale monolingual language models pre-trained for Vietnamese. Experimental results show that PhoBERT consistently outperforms the recent best pre-trained multilingual model XLM-R (Conneau et al., 2020) and improves the state-of-the-art in multiple Vietnamese-specific NLP tasks including Part-of-speech tagging, Dependency parsing, Named-entity recognition and Natural language inference. We release PhoBERT to facilitate future research and downstream applications for Vietnamese NLP. Our PhoBERT models are available at https://github.com/VinAIResearch/PhoBERT

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, methodology): “As one example, the pre-trained checkpoints for the “Text-to-Text Transfer Transformer” (T5) model released by Raffel et al. (2020) have been used to achieve state-of-the-art results on many benchmarks (Khashabi et al.”
FlauBERT: Unsupervised Language Model Pre-training for French (2019, background): “…Arabic (Antoun et al., 2020), Dutch (de Vries et al., 2019; Delobelle et al., 2020), Finnish (Virtanen et al., 2019), Italian (Polignano et al., 2019), Portuguese (Souza et al., 2019), Russian (Kuratov and Arkhipov, 2019), Spanish (Cañete et al., 2020), and Vietnamese (Nguyen and Nguyen, 2020).”
IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP (2020, methodology): “For dependency parsing, we follow Nguyen and Nguyen (2020) in incorporating BERT into the Bi-Afﬁne dependency parser (Dozat and Manning, 2017) by replacing the word embeddings with the corresponding contextualized representations.”
Language Models are Few-shot Multilingual Learners (2021, background): “Common approaches to solve the low-resource issue are to pretrain models with self-supervised learning using unlabelled monolingual text data collected from various resources available online (Wilie et al., 2020; Le et al., 2020; Martin et al., 2020; Eddine et al., 2020; Nguyen and Nguyen, 2020; Sch”
PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data (2020, background): “Recent works demonstrate that monolingual pretrained models perform better on tasks in that same language than models pretrained on multilingual corpora [20, 22, 11, 3, 23, 2, 14, 1, 8, 12].”
ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic (2020, methodology): “…outperformed by mono-lingual models pre-trained with larger vocabulary and bigger language-speciﬁc datasets (Virtanen et al., 2019; Antoun et al., 2020; Dadas et al., 2020; † All authors contributed equally. de Vries et al., 2019; Le et al., 2020; Martin et al., 2020; Nguyen and Tuan Nguyen, 2020).”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “trained based on standard model architectures like GPT [142] BERT [74], [75], [92], [133], [135], [136], [139]–[141], [144]–[146], [151]–[154], RoBERTa [137], [138], [147]– [149], [152], [155], ALBERT [92], [150], ELECTRA [132], [143], and T5 [77].”