Empirical Methods for Compound Splitting

By Philipp Koehn, Kevin Knight. 2003

380 citations recorded by the source corpus.

Compounded words are a challenge for NLP applications such as machine translation (MT). We introduce methods to learn splitting rules from monolingual and parallel corpora. We evaluate them against a gold standard and measure their impact on performance of statistical MT systems. Results show accuracy of 99.1% and performance gains for MT of 0.039 BLEU on a German-English noun phrase translation task.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Edinburgh’s Phrase-based Machine Translation Systems for WMT-14 (2014, methodology): “We used syntactic-preordering (Collins et al., 2005) and compound splitting (Koehn and Knight, 2003) for German-to-English systems.”
What do Neural Machine Translation Models Learn about Morphology? (2017, methodology): “Word representations in MT Machine translation systems that deal with morphologically-rich languages resort to various techniques for representing morphological knowledge, such as word segmentation (Nieflen and Ney, 2000; Koehn and Knight, 2003; Badr et al., 2008) and factored translation models (Ko”
Montreal Neural Machine Translation Systems for WMT’15 (2015, positive): “In the case of German (De) source, we performed compound splitting (Koehn and Knight, 2003), as implemented in the Moses toolkit (Koehn et al., 2007).”
Fast Domain Adaptation for Neural Machine Translation (2016, methodology): “The data is tokenized and the German text is preprocessed by splitting German compound words with the frequency-based method as described in [Koehn and Knight, 2003].”
EU-BRIDGE MT: Combined Machine Translation (2014, methodology): “Compound splitting (Koehn and Knight, 2003) is performed on the source side of the corpus for German → English translation before training.”
The Edinburgh/JHU Phrase-based Machine Translation Systems for WMT 2015 (2015, methodology): “For translation from German, we applied syntactic pre-reordering (Collins et al., 2005) and compound splitting (Koehn and Knight, 2003) in a preprocessing step on the source side.”
The Karlsruhe Institute of Technology Translation Systems for the WMT 2015 (2015, methodology): “For German → English translation, we apply compound splitting (Koehn and Knight, 2003) on the source side, in order to handle the out-of-vocabulary (OOV) issue of German compound words.”
The Karlsruhe Institute of Technology Translation Systems for the WMT 2014 (2014, background): “Compound splitting (Koehn and Knight, 2003) is performed on the source side of the corpus for German→English translation.”
Analyzing Parallelism and Domain Similarities in the MAREC Patent Corpus (2012, methodology): “We apply a simple empirical compound splitting method by [19], which is distributed as a script in the Moses toolkit.”
The RWTH Aachen German-English Machine Translation System for WMT 2015 (2015, methodology): “We reduced the source vocabulary size for the German→English translation and split the German compound words with the frequency-based method described in Koehn and Knight (2003). To reduce translation complexity, we employed the long-range part-of-speech based reordering rules proposed by Popović an”
The DCU-ICTCAS MT system at WMT 2014 on German-English Translation Task (2014, neutral): “And we conclude in the last section.”
Optimizing Chinese Word Segmentation for Machine Translation Performance (2008, background): “On different language pairs, (Koehn and Knight, 2003) and (Habash and Sadat, 2006) showed that data-driven methods for splitting and preprocessing can improve Arabic-English and German-English MT.”
The JHU Machine Translation Systems for WMT 2016 (2016, background): “and compound splitting (Koehn and Knight, 2003) for the German-to-English systems.”
YSDA Participation in the WMT’16 Quality Estimation Shared Task (2016, methodology): “…in German, we make a simple preprocessing by applying the following rules: • replace URLs and ﬁle names with a single dummy token • split German compounds with compound splitting algorithm similar to (Koehn and Knight, 2003) • remove redundant punctuation from provided machine translations”
English SLT and MT system description for the IWSLT 2013 evaluation (2013, background): “• source-side German compound splitting [23] • cube pruning with pop limit 1000 for tuning, 5000 for testing [24]”
The KIT-LIMSI Translation System for WMT 2014 (2014, background): “Compound splitting (Koehn and Knight, 2003) is performed on the source side (German) of the corpus before training.”
Adaptation of machine translation for multilingual information retrieval in the medical domain (2014, methodology): “based method for splitting compound words introduced by Koehn and Knight [45] and implemented in the script compound-splitter.”
Edinburgh’s Machine Translation Systems for European Language Pairs (2013, background): “…sets from 2008-2010, with a total of 7,567 sentences per language • German–English with syntactic pre-reordering (Collins et al., 2005), compound splitting (Koehn and Knight, 2003) and use of factored representation for a POS target sequence model (Koehn and Hoang, 2007) • English–German with…”
English-to-Czech Factored Machine Translation (2007, methodology): “…to employ morphology in improving word alignment techniques (e.g. (Popovi´c and Ney, 2004)) or machine translation quality (Nießen and Ney (2001), Koehn and Knight (2003), Zollmann et al. (2006), among others, for various languages; Goldwater and McClosky (2005), Bojar et al. (2006) and Talbot…”