Optimizing Chinese Word Segmentation for Machine Translation Performance

By Pi-Chuan Chang, Michel Galley, Christopher D. Manning. 2008

365 citations recorded by the source corpus.

Previous work has shown that Chinese word segmentation is useful for machine translation to English, yet the way different segmentation strategies affect MT is still poorly understood. In this paper, we demonstrate that optimizing segmentation for an existing segmentation standard does not always yield better MT performance. We find that other factors such as segmentation consistency and granularity of Chinese "words" can be more important for machine translation. Based on these findings, we implement methods inside a conditional random field segmenter that directly optimize segmentation granularity with respect to the MT task, providing an improvement of 0.73 BLEU. We also show that improving segmentation consistency using external lexicon and proper noun features yields a 0.32 BLEU increase.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Cross-lingual Language Model Pretraining (2019, methodology): “For Chinese, Japanese and Thai we use the tok-enizer of Chang et al. (2008), the Kytea 4 tokenizer, and the PyThaiNLP 5 tokenizer respectively.”
Learning Word Vectors for 157 Languages (2018, methodology): “We used the Stanford word segmenter (Chang et al., 2008) for Chinese, Mecab (Kudo, 2005) for Japanese and UETseg-menter (Nguyen and Le, 2016) for Vietnamese.”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, methodology): “XNLI consists of 7500 human-annotated development and test examples in NLI three-way classification format in English, French, Spanish, German, Greek, Bulgarian, Russian, Turkish, Arabic, Vietnamese, Thai, Chinese, Hindi, Swahili and Urdu, making a total of 112,500 annotated pairs.”
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks (2019, methodology): “For tokenization, we follows the line of Koehn et al. (2007); Chang et al. (2008) for each language.”
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback (2017, methodology): “For Chinese, we use the Stanford Chinese word segmenter (Chang et al., 2008) to segment sentences and tokenize.”
Cross-Lingual Natural Language Generation via Pre-Training (2019, methodology): “Following (Lample and Conneau 2019), we use the tokenizer provided by (Chang, Galley, and Manning 2008) for Chinese, and Moses1 for other languages, respectively.”
Self-Training PCFG Grammars with Latent Annotations Across Languages (2009, methodology): “We chose to use the Stanford segmenter (Chang et al., 2008) in our experiments because it is consistent with the treebank segmentation and provides the best performance among the segmenters that were tested.”
English SLT and MT system description for the IWSLT 2013 evaluation (2013, background): “• hierarchical lexicalized reordering (mslr) [31] • MADA tokenizer for source-side Arabic [32] • Stanford Chinese segmenter [33]”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, methodology): “The Chinese dataset is segmented with the Stanford word segmenter [61, 323].”
Machine Translation without Words through Substring Alignment (2012, background): “…works have attempted to handle morphology, decompounding and regularization through lemmatization, morphological analysis, or unsupervised techniques (Nießen and Ney, 2000; Brown, 2002; Lee, 2004; Goldwater and McClosky, 2005; Talbot and Osborne, 2006; Mermer and Akın, 2010; Macherey et al., 2011).”
Phrase Dependency Machine Translation with Quasi-Synchronous Tree-to-Tree Features (2014, neutral): “…the longest from ylk to y j i: maxu:i≤u≤j,k≤τy (u)≤l |τy (u)− u| > max v:k≤v≤l,i≤τy (v)≤j |τy (v)− v| The final condition also implies that there is a lexical dependency from a word in yji to a word in ylk: ∃u, i ≤ u ≤ j, such that k ≤ τy (u) ≤ l. 7 Other ways of getting phrase dependencies are…”
Yandex School of Data Analysis approach to English-Turkish translation at WMT16 News Translation Task (2016, background): “• An excessive segmentation does not really hurt a phrase-based system, as shown by (Chang et al., 2008).”
A Systematic Exploration of Diversity in Machine Translation (2013, methodology): “We segmented the Chinese data using the Stanford Chinese segmenter (Chang et al., 2008) in “CTB” mode, giving us 7.”
Fast Online Lexicon Learning for Grounded Language Acquisition (2012, methodology): “In this paper we presented a novel online algorithm for building a lexicon from ambiguously supervised relational data.”