Automatic Tagging of Arabic Text: From Raw Text to Base Phrase Chunks

By Mona T. Diab, Kadri Hacioglu, Dan Jurafsky. North American Chapter of the Association for Computational Linguistics, 2004

403 citations recorded by the source corpus.

To date, there are no fully automated systems addressing the community's need for fundamental language processing tools for Arabic text. In this paper, we present a Support Vector Machine (SVM) based approach to automatically tokenize (segmenting off clitics), part-of-speech (POS) tag and annotate base phrases (BPs) in Arabic text. We adapt highly accurate tools that have been developed for English text and apply them to Arabic text. Using standard evaluation metrics, we report that the SVM-TOK tokenizer achieves an Fβ=1 score of 99.12, the SVM-POS tagger achieves an accuracy of 95.49%, and the SVM-BP chunker yields an Fβ=1 score of 92.08.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/c59826bcf67cac24d8024055b1c33251f7edf3c9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Natural language processing: state of the art, current trends and challenges (2017, methodology): “Diab et al. (2004) [34] used supervised machine learning approach and adopted Support Vector Machines (SVMs) which were trained on the Arabic Treebank to automatically tokenize parts of speech tag and annotate base phrases in Arabic text.”
Context-Based Morphological Disambiguation with Random Fields (2005, background): “Both Diab et al. (2004) and Habash and Rambow (2005) use support-vector machines with local features; the former for tokenization, POS tagging, and base phrase chunking; the latter for full morphological disambiguation.”
WIT3: Web Inventory of Transcribed and Translated Talks (2012, methodology): “Different preprocessings were performed depending to the language: Arabic and Chinese were segmented by means of AMIRA (Diab et al., 2004) and the Stanford Chinese Segmenter (Tseng et al., 2005), respectively; the tokenizer script released together with the Europarl corpus (Koehn, 2005) was applied ”
Learning Bilingual Lexicons from Monolingual Corpora (2008, methodology): “(11)We use the Tree Tagger (Schmid, 1994) for all POS tagging except for Arabic, where we use the tagger described in Diab et al. (2004).”
The IWSLT 2011 Evaluation Campaign on Automatic Talk Translation (2012, methodology): “…computing statistics with the following tools: for English and French, the tokenizer script released together with Europarl corpus (Koehn, 2005); for Arabic, the AMIRA segmenter (M. Diab and Jurafsky, 2004) (Arianna Bisazza and Federico, 2010); for Mandarin the Standford segmenter (Manning, 2002).”
Challenging Language-Dependent Segmentation for Arabic: An Application to Machine Translation and Part-of-Speech Tagging (2017, background): “Arabic word segmentation has shown to signiﬁ-cantly improve output quality in NLP tasks such as machine translation (Habash and Sadat, 2006; Almahairi et al., 2016), part-of-speech tagging (Diab et al., 2004; Habash and Rambow, 2005), and information retrieval (M. Aljlayl and Gross-man, 2002).”
Book Reviews: Introduction to Arabic Natural Language Processing by Nizar Y. Habash (2010, methodology): “2) [9, 109].”
Arabic Tokenization System (2007, methodology): “These solutions include morphological analysis (Beesley 2001; Buckwalter 2002), diacritization (Nelken and Shieber 2005), Information Retrieval (Larkey and Connell 2002), and POS Tagging (Diab et al 2004; Habash and Rambow 2005).”
An Unsupervised Morpheme-Based HMM for Hebrew Morphological Disambiguation (2006, background): “As reported by Habash and Rambow, the (cid:12)rst work on Arabic tagging which used a corpus for training and evaluation was the work of Diab et al. (2004).”
Identifying Broken Plurals, Irregular Gender, and Rationality in Arabic Text (2012, methodology): “…tools for Arabic NLP: the Buckwalter Arabic Morphological Analyzer (BAMA) (Buckwalter, 2004) which is used in the Penn Arabic Tree Bank (PATB) (Maamouri et al., 2004), and the various POS tagging and morphological disambiguation tools trained using them (Diab et al., 2004; Habash and Rambow, 2005).”
FBK@IWSLT 2011 (2011, methodology): “The Arabic text is then segmented with AMIRA [16] according to the Arabic Treebank (ATB) scheme that isolates conjunctions w+ and f+, prepositions l+, k+, b+, future marker s+, pronominal suffixes, but not the article Al+.”
Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop (2005, methodology): “The only work on Arabic tagging that uses a corpus for training and evaluation (that we are aware of), (Diab et al., 2004), does not use a morphological analyzer.”