Chinese Segmentation and New Word Detection using Conditional Random Fields

By Fuchun Peng, Fangfang Feng, Andrew McCallum. International Conference on Computational Linguistics, 2004

520 citations recorded by the source corpus.

Chinese word segmentation is a difficult, important and widely-studied sequence modeling problem. This paper demonstrates the ability of linear-chain conditional random fields (CRFs) to perform robust and accurate Chinese word segmentation by providing a principled framework that easily supports the integration of domain knowledge in the form of multiple lexicons of characters and words. We also present a probabilistic new word detection method, which further improves performance. Our system is evaluated on four datasets used in a recent comprehensive Chinese word segmentation competition. State-of-the-art performance is obtained.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/929f9332970683c9bcccde8fe49fabcd2ae687a6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Optimizing Chinese Word Segmentation for Machine Translation Performance (2008, methodology): “CRF is a statistical sequence modeling framework introduced by Lafferty et al. (2001), and was ﬁrst used for the Chinese word segmentation task by Peng et al. (2004), who treated word segmentation as a binary decision task.”
An Introduction to Conditional Random Fields (2010, background): “Since then, linear-chain CRFs have been applied to many problems in natural language processing, including named-entity recognition [72], feature induction for NER [71], shallow parsing [108, 120], identifying protein names in biology abstracts [107], segmenting addresses in Web pages [26], informat”
A review of natural language processing techniques for opinion mining systems (2017, methodology): “Conditional Random Fields (CRFs) [43] have been applied to this problem and outperformed hidden Markov models and maximum-entropy Markov models [44, 45, 46].”
Syntactic Processing Using the Generalized Perceptron and Beam Search (2011, methodology): “We follow the format from Peng, Feng, and McCallum (2004), where each row represents a CWS model.”
A conditional random field-based model for joint sequence segmentation and classification (2013, background): “During the last years, we have witnessed an explosion of interest in CRFs, as they have managed to achieve superb prediction performance in a variety of scenarios, thus being one of the most successful approaches to the structured output prediction problem, with successful applications including tex”
Don’t Parse, Generate! A Sequence to Sequence Architecture for Task-Oriented Semantic Parsing (2020, methodology): “CRFs learn pairwise potentials on labeling subsequent words which allow models to find more probable label sequences for a given query.”
Context-Based Morphological Disambiguation with Random Fields (2005, methodology): “When training a CRF for POS tagging, IOB chunking (Sha and Pereira, 2003), or word segmentation (Peng et al., 2004), one typically structures the conditional probabilities (in the objective function) using domain knowledge: in POS tagging, the set of allowed tags for a word is used; in IOB…”
A Conditional Random Field Word Segmenter for Sighan Bakeoff 2005 (2005, methodology): “As seen in Table 2, our system’s F-score was 0.909 on CTB (Chinese Treebank from University of Pennsylvania) versus 0.849 F on Peng et al. (2004).”
Neural Word Segmentation with Rich Pretraining (2017, methodology): “State-of-the-art approaches include character sequence labeling models (Xue et al., 2003) using CRFs (Peng et al., 2004; Zhao et al., 2006) and max-margin structured models leveraging word features (Zhang and Clark, 2007; Sun et al., 2009; Sun, 2010).”
Semi-Supervised Learning for Natural Language (2005, methodology): “(Peng et al., 2004) also treats CWS as a tagging problem but uses a Conditional Random Field (Section 3.2.5), which generally outperforms maximum entropy models (Lafferty et al., 2001a).”
Applying Conditional Random Fields to Japanese Morphological Analysis (2004, methodology): “…CRFs have been reported recently in part-of-speech tagging (Lafferty et al., 2001), shallow parsing (Sha and Pereira, 2003), named entity recognition (Mc-Callum and Li, 2003), Chinese word segmentation (Peng et al., 2004), and Information Extraction (Pinto et al., 2003; Peng and McCallum, 2004).”
Pointwise Prediction for Robust, Adaptable Japanese Morphological Analysis (2011, methodology): “Previous work using this two-stage approach has used sequence-based prediction methods, such as maximum entropy Markov models (MEMMs) or CRFs (Ng and Low, 2004; Peng et al., 2004).”
An Introduction to Conditional Random Fields for Relational Learning (2007, methodology): “…2005], segmenting addresses in Web pages [Culotta et al., 2004], ﬁnding semantic roles in text [Roth and Yih, 2005], identifying the sources of opinions [Choi et al., 2005], Chinese word segmentation [Peng et al., 2004], Japanese morphological analysis [Kudo et al., 2004], and many others.”
Effects of Semantic Features on Machine Learning-Based Drug Name Recognition Systems: Word Embeddings vs. Manually Constructed Dictionaries (2015, background): “It has been widely applied to a large number of Natural Language Processing (NLP) tasks such as named entity recognition [45], shallow parsing [46], and Chinese word segmentation [47] and has shown good performances on these tasks.”