AraBERT: Transformer-based Model for Arabic Language Understanding

By Wissam Antoun, Fady Baly, Hazem M. Hajj. OSACT, 2020

1,444 citations recorded by the source corpus.

The Arabic language is a morphologically rich language with relatively few resources and a less explored syntax compared to English. Given these limitations, Arabic Natural Language Processing (NLP) tasks like Sentiment Analysis (SA), Named Entity Recognition (NER), and Question Answering (QA), have proven to be very challenging to tackle. Recently, with the surge of transformers based models, language-specific BERT based models have proven to be very efficient at language understanding, provided they are pre-trained on a very large corpus. Such models were able to set new standards and achieve state-of-the-art results for most NLP tasks. In this paper, we pre-trained BERT specifically for the Arabic language in the pursuit of achieving the same success that BERT did for the English language. The performance of AraBERT is compared to multilingual BERT from Google and other state-of-the-art approaches. The results showed that the newly developed AraBERT achieved state-of-the-art performance on most tested Arabic NLP tasks. The pretrained araBERT models are publicly available on https://github.com/aub-mind/araBERT hoping to encourage research and applications for Arabic NLP.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/1359d2ef45f1550941e22bf046026c89f6edf315

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic (2020, methodology): “Our models compare to mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020) (base and large), and AraBERT (Antoun et al., 2020) in terms of training data size, vocabulary size, and overall model capacity as we summarize in Table 2.”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, neutral): “BioBERT [91] FT PubMed, PMC Biomedical NER, RE, QA BlueBert [130] FT PubMed, MIMIC-III BLUE MIMIC-BERT [131] FT MIMIC-III Biomedical NER BioFLAIR [132] FT PubMed Bio NER Bio-ELECTRA-small [133] PT PubMed Biomedical NER AlphaBERT [134] FT Discharge diagnoses Extractive Summarization Task Spanish-bert”
Masader: Metadata Sourcing for Arabic Text and Speech Data Resources (2021, neutral): “We also anticipate a significant increase, particularly in 2020, with the emergence of pretrained language models namely AraBERT (Antoun et al., 2020a), Multi-dialect BERT (Talafha et al.”
Probing Toxic Content in Large Pre-Trained Language Models (2021, methodology): “…we study the same phenomenon in two other languages by translating more than 700 ATOMIC commonsense actions to Arabic and French, along with names of social groups, then run the same experiments using the French PTLM CamemBERT (Martin et al., 2020), and the Arabic AraBERT (Antoun et al., 2020).”
Challenges and Strategies in Cross-Cultural NLP (2022, background): “A PLM speciﬁc to the Algerian dialect of Arabic performs better than a multilingual or general Arabic one (Antoun et al., 2020) in sentiment classiﬁcation (Abdaoui et al., 2021).”
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, methodology): “For instance, there are monolingual BERT variants for Arabic (Antoun et al., 2020), French (Martin et al., 2020), Finnish (Virtanen et al., 2019), Dutch (de Vries et al., 2019), Italian (Polignano et al., 2019), to name only a few.”
FlauBERT: Unsupervised Language Model Pre-training for French (2019, methodology): “Arabic word sense disambiguation for and by machine translation.”
Telling BERT’s Full Story: from Local Attention to Global Aggregation (2020, methodology): “To understand whether this behavior is particular to English we perform the same calculations on the German (Deepset, 2019) and Arabic (Antoun et al., 2020) versions of BERT on 1800 examples of the XNLI dataset (Conneau et al., 2018); note that although Arabic is a right-to-left language, to process”
The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models (2021, background): “2 (Antoun et al., 2020) MSA 77GB 8.6B WP 60k 3M X 4 ArabicBERT (Safaya et al., 2020) MSA 95GB 8.2B WP 32k 4M X 5 Multi-dialect-Arabic-BERT MSA/DA - - WP 32k - (Talafha et al., 2020) X 6 GigaBERTv4 (Lan et al., 2020) MSA - 10.4B WP 50k 1.48M X 7 MARBERT (Abdul-Mageed et al., 2020a)”