Spanish Pre-trained BERT Model and Evaluation Data
By J. Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, Jorge P'erez. arXiv.org, 2023
793 citations recorded by the source corpus.
The Spanish language is one of the top 5 spoken languages in the world. Nevertheless, finding resources to train or evaluate Spanish language models is not an easy task. In this paper we help bridge this gap by presenting a BERT-based language model pre-trained exclusively on Spanish data. As a second contribution, we also compiled several tasks specifically for the Spanish language in a single repository much in the spirit of the GLUE benchmark. By fine-tuning our pre-trained Spanish model, we obtain better results compared to other BERT-based models pre-trained on multilingual corpora for most of the tasks, even achieving a new state-of-the-art on some of them. We have publicly released our model, the pre-training data, and the compilation of the Spanish benchmarks.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/79926aa63d4daee6af06a8e9a7c2480b31cb7ed9
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Tahmid Hasan
- Ajit Rajasekharan
- Phillip Rust
- Anders Søgaard
- Anindya Iqbal
- Kazi Samin Mubasshir
- Rifat Shahriyar
- Daniel Hershcovich
Citations in the corpus
- AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “trained based on standard model architectures like GPT [142] BERT [74], [75], [92], [133], [135], [136], [139]–[141], [144]–[146], [151]–[154], RoBERTa [137], [138], [147]– [149], [152], [155], ALBERT [92], [150], ELECTRA [132], [143], and T5 [77].”
- Challenges and Strategies in Cross-Cultural NLP (2022, background): “For example, there are considerable morphosyntactic variations between Spanish spoken in Spain and Argentina (Bentivoglio and Sedano, 2011), but they are not considered separately in a Spanish PLM (Cañete et al., 2020).”
- BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla (2021, methodology): “Motivated by the triumph of language-specific models (Martin et al. (2020); Polignano et al. (2019); Canete et al. (2020); An-toun et al. (2020), inter alia ) over multilingual models in many other languages, in this work, we present BanglaBERT – a BERT-based (Devlin et al., 2019) Bangla NLU model…”