AlBERTo: Italian BERT Language Understanding Model for NLP Challenging Tasks Based on Tweets

By Marco Polignano, Pierpaolo Basile, Marco Degemmis, Giovanni Semeraro, Valerio Basile. 2019

235 citations recorded by the source corpus.

Subfields: Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Natural Language Processing

Citations in the corpus

mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, background): “…has been to release dozens of models that have instead been pre-trained on a single non-English language (Carmo et al., 2020; de Vries et al., 2019; Le et al., 2019; Martin et al., 2019; Delobelle et al., 2020; Malmsten et al., 2020; Nguyen and Nguyen, 2020; Polignano et al., 2019, etc.).”
FlauBERT: Unsupervised Language Model Pre-training for French (2019, background): “In Proceedings of the Sixth Italian Conference on Computational Linguistics (CLiC-it 2019), volume 2481.”
Spanish Pre-trained BERT Model and Evaluation Data (2023, background): “This has led to the release ∗Work partially performed while at Adereso. of BERT models in Russian (Kuratov & Arkhipov, 2019), French (Martin et al., 2019; Le et al., 2019), Dutch (de Vries et al., 2019; Delobelle et al., 2020), Italian (Polignano et al., 2019), and Portugese (Souza et al., 2019).”
PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data (2020, background): “Recent works demonstrate that monolingual pretrained models perform better on tasks in that same language than models pretrained on multilingual corpora [20, 22, 11, 3, 23, 2, 14, 1, 8, 12].”
ETHOS: a multi-label hate speech detection dataset (2020, background): “Such architectures have met great acceptance in the related ML community [36, 58].”
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, background): “For instance, there are monolingual BERT variants for Arabic (Antoun et al., 2020), French (Martin et al., 2020), Finnish (Virtanen et al., 2019), Dutch (de Vries et al., 2019), Italian (Polignano et al., 2019), to name only a few.”
Time of Your Hate: The Challenge of Time in Hate Speech Detection on Social Media (2020, methodology): “More details about AlBERTo are available in [21,23,43].”
BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla (2021, methodology): “Motivated by the triumph of language-speciﬁc models (Martin et al. (2020); Polignano et al. (2019); Canete et al. (2020); An-toun et al. (2020), inter alia ) over multilingual models in many other languages, in this work, we present BanglaBERT – a BERT-based (Devlin et al., 2019) Bangla NLU model…”
AraBERT: Transformer-based Model for Arabic Language Understanding (2020, background): “…a multilingual model was provided for the public, pre-training monolingual BERT for other languages proved to provide better performance over the multilingual BERT such as Italian BERT Al-berto (Polignano et al., 2019) and other publicly available BERTs (Martin et al., 2019; de Vries et al., 2019).”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “trained based on standard model architectures like GPT [142] BERT [74], [75], [92], [133], [135], [136], [139]–[141], [144]–[146], [151]–[154], RoBERTa [137], [138], [147]– [149], [152], [155], ALBERT [92], [150], ELECTRA [132], [143], and T5 [77].”
BERTje: A Dutch BERT Model (2019, methodology): “architecture have been developed for Italian (Polignano et al., 2019), French (Le et al.”
Overview of the Transformer-based Models for NLP Tasks (2020, background): “For example BERTje [36] is a Dutch version, AlBERTo [37] is an Italian version, and CamemBERT [38] and FlauBERT [39] are two different models for French.”