A Conditional Random Field Word Segmenter for Sighan Bakeoff 2005

By Huihsin Tseng, Pi-Chuan Chang, Galen Andrew, Dan Jurafsky, Christopher D. Manning. International Joint Conference on Natural Language Processing, 2005

522 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/c9f46f7e2f43f0b38a31007aebb78f94c6f146c8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Text Classification Algorithms: A Survey (2019, background): “with words that were not present in training data sample) [176].”
Overview of the 5th Workshop on Asian Translation (2019, methodology): “For Japanese segmentation, we used three different tools: Juman version 7.0 (Kurohashi et al., 1994), KyTea 0.4.6 (Neubig et al., 2011) with full SVM model23 and MeCab 0.996 (Kudo, 2005) with IPA dictionary 2.7.0.24 For Chinese segmentation, we used two different tools: KyTea 0.4.6 with full SVM Mod”
Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context (2017, methodology): “Stanford Segmenter (Tseng et al., 2005) was used to pre-process the Chinese corpora.”
Cross-lingual Models of Word Embeddings: An Empirical Comparison (2016, methodology): “We use the Stanford Chinese Segmenter (Tseng et al., 2005) to preprocess the en-zh parallel corpus.”
WIT3: Web Inventory of Transcribed and Translated Talks (2012, methodology): “…were performed depending to the language: Arabic and Chinese were segmented by means of AMIRA (Diab et al., 2004) and the Stanford Chinese Segmenter (Tseng et al., 2005), respectively; the tokenizer script released together with the Europarl corpus (Koehn, 2005) was applied to other languages.”
The IWSLT 2011 Evaluation Campaign on Automatic Talk Translation (2012, methodology): “…computing statistics with the following tools: for English and French, the tokenizer script released together with Europarl corpus (Koehn, 2005); for Arabic, the AMIRA segmenter (M. Diab and Jurafsky, 2004) (Arianna Bisazza and Federico, 2010); for Mandarin the Standford segmenter (Manning, 2002).”
Overview of the 1st Workshop on Asian Translation (2014, methodology): “6 with Full SVM Model in MSR model and Stanford Word Segmenter version 2014-06-16 with Chinese Penn Treebank (CTB) and Peking University (PKU) model 9 (Tseng, 2005).”
A review of natural language processing techniques for opinion mining systems (2017, methodology): “Conditional Random Fields (CRFs) [43] have been applied to this problem and outperformed hidden Markov models and maximum-entropy Markov models [44, 45, 46].”
The IWSLT 2015 Evaluation Campaign (2015, methodology): “The standard tokenization via the tokenizer script released with the Europarl corpus [31] was applied to all languages, with the exception of Chinese and Thai; the former was preprocessed by means of the Stanford Chinese Segmenter [32], while the Thai texts were segmented according to the guidelines”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, methodology): “The Chinese dataset is segmented with the Stanford word segmenter [61, 323].”
Optimizing Chinese Word Segmentation for Machine Translation Performance (2008, methodology): “Our linguistic features are adopted from (Ng and Low, 2004) and (Tseng et al., 2005).”
Neural Word Segmentation with Rich Pretraining (2017, methodology): “For pretraining character representations, we extract punctuation classification data from the Gigaword corpus, and use the word-based ZPar and a standard character-based CRF model (Tseng et al., 2005) to obtain automatic segmentation results.”
Multi-Domain Neural Machine Translation with Word-Level Adaptive Layer-wise Domain Mixing (2019, methodology): “We tokenize English, German and French sentences using MOSES script (Koehn et al., 2007) and perform word segmentation on Chinese sentences using Stanford Segmenter (Tseng et al., 2005).”
Linguistically-augmented perplexity-based data selection for language models (2015, methodology): “…(word segmentation, PoS tagging and NE recognition) with the Stanford CoreNLP toolkit.14 From this toolkit we have used a CRF-based word segmenter (Tseng, 2005; Chang et al., 2008), a maximum-entropy PoS tagger (Toutanova et al., 2003) and a CRF-based NE recogniser (Finkel et al., 2005) with…”
Max-Margin Tensor Neural Network for Chinese Word Segmentation (2014, background): “The most popular approach treats word segmentation as a sequence labeling problem which was first proposed in Xue (2003). Most previous systems address this task by using linear statistical models with carefully designed features such as bigram features, punctuation information (Li and Sun, 2009) an”
Report on the 11th IWSLT evaluation campaign (2014, methodology): “preprocessed by, respectively: the Stanford Chinese Segmenter [34] and the QCRI-normalizer.”