Neural Word Segmentation with Rich Pretraining

By Jie Yang, Fei Dong. 2017

117 citations recorded by the source corpus.

Neural word segmentation research has benefited from large-scale raw texts by leveraging them for pretraining character and word embeddings. On the other hand, statistical segmentation research has exploited richer sources of external information, such as punctuation, automatic segmentation and POS. We investigate the effectiveness of a range of external training sources for neural word segmentation by building a modular segmentation model, pretraining the most important submodule using rich external sources. Results show that such pretraining significantly improves the model, leading to accuracies competitive to the best methods on six benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Noah A. Smith
2. Sebastian Ruder
3. Yoav Goldberg
4. Luke S. Zettlemoyer
5. Mohit Bansal
6. Christopher D. Manning
7. Swabha Swayamdipta
8. Graham Neubig

In Computer Vision and Pattern Recognition

1. Sebastian Ruder
2. Thomas Wolf
3. Matthew E. Peters
4. Jan A. Botha
5. Swabha Swayamdipta
6. Ryan T. McDonald
7. David Weiss
8. Jie Yang

In Natural Language Processing

1. Sebastian Ruder
2. Noah A. Smith
3. Thomas Wolf
4. Matthew E. Peters
5. Swabha Swayamdipta
6. Yoav Goldberg
7. Mohit Bansal
8. Luke S. Zettlemoyer

Citations in the corpus

• Transfer Learning in Natural Language Processing (2019, methodology): “We will then describe supervised approaches which make use of large annotated datasets (Zoph et al., 2016; Yang et al., 2017; Wieting et al., 2016; Conneau et al., 2017; McCann et al., 2017) before turning to distant supervision approaches which use heuristics to automatically label datasets (Mintz ”
• Natural Language Processing with Small Feed-Forward Networks (2017, background): “Recently, neural networks have significantly improved segmentation accuracy (Zhang et al., 2016; Cai and Zhao, 2016; Liu et al., 2016; Yang et al., 2017; Kong et al., 2015).”
• Chinese NER Using Lattice LSTM (2018, methodology): “For Weibo and resume, we take the best model of Yang et al. (2017a) off the shelf5, 3https://www.weibo.com/ 4http://finance.sina.com.cn/stock/ index.shtml 5https://github.com/jiesutd/ RichWordSegmentor which is trained using CTB 6.0 (Xue et al., 2005).”