Unsupervised Discovery of Morphemes

By Mathias Creutz, Krista Lagus. 2002

423 citations recorded by the source corpus.

We present two methods for unsupervised segmentation of words into morpheme-like units. The model utilized is especially suited for languages with a rich morphology, such as Finnish. The first method is based on the Minimum Description Length (MDL) principle and works online. In the second method, Maximum Likelihood (ML) optimization is used. The quality of the segmentations is measured using an evaluation method that compares the segmentations produced to an existing morphological analysis. Experiments on both Finnish and English corpora show that the presented methods perform well compared to a current state-of-the-art system.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Morphology-based and sub-word language modeling for Turkish speech recognition (2010, methodology): “The baseline algorithm introduced in [9] is used to segment word types in the text corpus.”
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, neutral): “0 (Creutz and Lagus, 2002), later called the Morfessor Baseline,”
Morfessor 2.0: Python Implementation and Extensions for Morfessor Baseline (2013, background): “The most popular versions of Morfessor are Morfessor Baseline (Creutz and Lagus, 2002, 2005b) and Morfessor Categories-MAP (Creutz and Lagus, 2005a, 2007).”
Hybrid Morphological Segmentation for Phrase-Based Machine Translation (2016, methodology): “In order to force the Morfessor method to follow the linguistic morphs produced by Omorfi, we added some new features to the Morfessor Baseline implementation by Virpioja et al. (2013).”
Unsupervised Morpheme Segmentation and Morphology Induction from Text Corpora Using Morfessor 1.0 (2005, methodology): “The models presented in Creutz and Lagus (2002) and Creutz (2003) differ in the way they assign probabilities to different morph length and frequency values. Version 1.0 of the Morfessor program supports both previous models, slightly modified. What we here call the Morfessor Baseline model is based”
A Bayesian framework for word segmentation: exploring the effects of context. (2009, positive): “MDL is a particular formulation of Bayesian learning that has been used successfully in a number of other areas of language acquisition as well (Ellison, 1994; Goldsmith, 2001; Goldwater and Johnson, 2004; Creutz and Lagus, 2002; Dowman, 2000).”
Morfessor in the Morpho Challenge (2006, methodology): “The Morfessor Baseline algorithm was originally introduced in (Creutz and Lagus, 2002), where it was called the “Recursive MDL” method.”
Collecting Resources in Sub-Saharan African Languages for Automatic Speech Recognition: a Case Study of Wolof (2016, methodology): “Concerning the language model, it was created using SRILM using 3-grams and the text is segmented in morphemes using Morfessor 2.0 (Creutz and Lagus, 2002).”