Arap-Tweet: A Large Multi-Dialect Twitter Corpus for Gender, Age and Language Variety Identification

By W. Zaghouani, A. Charfi. 2018

77 citations recorded by the source corpus.

In this paper, we present Arap-Tweet, which is a large-scale and multi-dialectal corpus of Tweets from 11 regions and 16 countries in the Arab world representing the major Arabic dialectal varieties. To build this corpus, we collected data from Twitter and we provided a team of experienced annotators with annotation guidelines that they used to annotate the corpus for age categories, gender, and dialectal variety. During the data collection effort, we based our search on distinctive keywords that are specific to the different Arabic dialects and we also validated the location using Twitter API. In this paper, we report on the corpus data collection and annotation efforts. We also present some issues that we encountered during these phases. Then, we present the results of the evaluation performed to ensure the consistency of the annotation. The provided corpus will enrich the limited set of available language resources for Arabic and will be an invaluable enabler for developing author profiling tools and NLP tools for Arabic.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Toward Gender-Inclusive Coreference Resolution: An Analysis of Gender and Bias Throughout the Machine Learning Lifecycle* (2021, result): “Second, similar to Zhao et al. (2018a) and Rudinger et al.”
The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models (2021, unknown): “DA Training Data For DA, we collect a range of dialectal corpora: LDC97T19-CALLHOME Transcripts (Gadalla et al., 1997); LDC2002T38-CALLHOME Supplement Transcripts (Linguistic Data Consortium, 2002); LDC2005S08-Babylon Levantine Arabic Transcripts (BBN Technologies, 2005); LDC2005S14-CTS Levantine Ar”
The MADAR Arabic Dialect Corpus and Lexicon (2018, background): “Various other research work resulted in multidialectal non parallel corpora at different scales (Zaidan and Callison-Burch, 2011; Zbib et al., 2012; Cotterell and Callison-Burch, 2014; Salama et al., 2014; Jeblee et al., 2014; Al-Shargi et al., 2016; Zaghouani and Charfi, 2018).”
ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic (2020, methodology): “For both age and gender, we use Arap-Tweet (Zaghouani and Charﬁ, 2018).”