Distributional Representations for Handling Sparsity in Supervised Sequence-Labeling

By Fei Huang, Alexander Yates. 2009

100 citations recorded by the source corpus.

Supervised sequence-labeling systems in natural language processing often suffer from data sparsity because they use word types as features in their prediction tasks. Consequently, they have difficulty estimating parameters for types which appear in the test set, but seldom (or never) appear in the training set. We demonstrate that distributional representations of word types, trained on unannotated text, can be used to improve performance on rare words. We incorporate aspects of these representations into the feature space of our sequence-labeling systems. In an experiment on a standard chunking dataset, our best technique improves a chunker from 0.76 F1 to 0.86 F1 on chunks beginning with rare words. On the same dataset, it improves our part-of-speech tagger from 74% to 80% accuracy on rare words. Furthermore, our system improves significantly over a baseline system when applied to text from a different domain, and it reduces the sample complexity of sequence labeling.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Information Extraction

Citations in the corpus

Natural Language Processing (Almost) from Scratch (2011, methodology): “Finally, Huang and Yates (2009) have recently proposed a smoothed language modelling approach based on a Hidden Markov Model, with success on POS and Chunking tasks.”
Word Representations: A Simple and General Method for Semi-Supervised Learning (2010, methodology): “This technique was used by a variety of researchers (Miller et al., 2004; Liang, 2005; Koo et al., 2008; Ratinov & Roth, 2009; Huang & Yates, 2009).”
Transfer learning using computational intelligence: A survey (2015, background): “The feature representation approaches can be categorized into two classes [11]: (A) Distribution similarity approaches aim explicitly to make the source and target domain sample distributions similar, either by penalizing or removing features whose statistics vary between domains [24, 30-32] or by l”
Domain Adaptation with Coupled Subspaces (2011, methodology): “The right-hand side of Figure 1 is an example of a part of speech (PoS) tagging task [31, 8, 19].”