Arabic Tokenization System

By Mohammed Attia. 2007

84 citations recorded by the source corpus.

Tokenization is a necessary and non-trivial step in natural language processing. In the case of Arabic, where a single word can comprise up to four independent tokens, morphological knowledge needs to be incorporated into the tokenizer. In this paper we describe a rule-based tokenizer that handles tokenization as a full-rounded process with a preprocessing stage (white space normalizer), and a post-processing stage (token filter). We also show how it handles multiword expressions, and how ambiguity is resolved.

Subfields: Artificial Intelligence, Natural Language Processing, Computational Linguistics & Semantics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ali Farghaly
2. Khaled Shaalan

In Natural Language Processing

1. Ali Farghaly
2. Khaled Shaalan

In Computational Linguistics & Semantics

1. Ali Farghaly
2. Khaled Shaalan

Citations in the corpus

• TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, neutral): “Clitics are common (Attia, 2007).”
• Arabic Natural Language Processing: Challenges and Solutions (2009, background): “Thus the complex internal structure of Arabic words makes tokenization, usually one of the early preprocessing tasks of a text, very challenging [Attia 2007].”