Context-Based Morphological Disambiguation with Random Fields
By Noah A. Smith, David A. Smith, Roy W. Tromble. 2005
66 citations recorded by the source corpus.
Finite-state approaches have been highly successful at describing the morphological processes of many languages. Such approaches have largely focused on modeling the phone- or character-level processes that generate candidate lexical types, rather than tokens in context. For the full analysis of words in context, disambiguation is also required (Hakkani-Tur et al., 2000; Hajic et al., 2001). In this paper, we apply a novel source-channel model to the problem of morphological disambiguation (segmentation into morphemes, lemmatization, and POS tagging) for concatenative, templatic, and inflectional languages. The channel model exploits an existing morphological dictionary, constraining each word's analysis to be linguistically valid. The source model is a factored, conditionally-estimated random field (Lafferty et al., 2001) that learns to disambiguate the full sentence by modeling local contexts. Compared with baseline state-of-the-art methods, our method achieves statistically significant error rate reductions on Korean, Arabic, and Czech, for various training set sizes and accuracy measures.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Nizar Habash
- Hinrich Schütze
- Yoav Goldberg
- Michael Elhadad
- Helmut Schmid
- Yonatan Belinkov
- Timo Schick
- Owen Rambow
Citations in the corpus
- Efficient Higher-Order CRFs for Morphological Tagging (2013, methodology): “Smith et al. (2005) use CRFs for POS+MORPH tagging, but use a morphological analyzer for candidate selection.”
- MADA + TOKAN : A Toolkit for Arabic Tokenization , Diacritization , Morphological Disambiguation , POS Tagging , Stemming and Lemmatization (2009, methodology): “Smith et al. (2005) presented a very similar setup to MADA’s first version (Habash and Rambow, 2005), except for using Context Random Fields instead of Support Vector Machines.”
- Word Segmentation, Unknown-word Resolution, and Morphological Agreement in a Hebrew Parsing System (2013, methodology): “It is also used for Arabic (Green and Manning 2010) and other languages (Smith, Smith, and Tromble 2005).”
- Book Reviews: Introduction to Arabic Natural Language Processing by Nizar Y. Habash (2010, background): “Much work has happened on this front [113,109,51,114,115,116,117].”