Robust Named Entity Recognition with Truecasing Pretraining

By Stephen Mayhew, Nitish Gupta, Dan Roth. 2019

43 citations recorded by the source corpus.

Although modern named entity recognition (NER) systems show impressive performance on standard datasets, they perform poorly when presented with noisy data. In particular, capitalization is a strong signal for entities in many languages, and even state of the art models overfit to this feature, with drastically lower performance on uncapitalized text. In this work, we address the problem of robustness of NER systems in data with noisy or uncertain casing, using a pretraining objective that predicts casing in text, or a truecaser, leveraging unlabeled data. The pretrained truecaser is combined with a standard BiLSTM-CRF model for NER by appending output distributions to character embeddings. In experiments over several datasets of varying domain and casing quality, we show that our new model improves performance in uncased text, even adding value to uncased BERT embeddings. Our method achieves a new state of the art on the WNUT17 shared task dataset.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Few-Shot Text Generation with Natural Language Instructions (2020, background): “Examples include casing prediction for named entity recognition (Mayhew et al., 2020), gap sentence generation for summarization (Zhang et al.”
Few-Shot Text Generation with Pattern-Exploiting Training (2020, background): “Examples include casing prediction for named entity recognition (Mayhew et al., 2020), gap sentence generation for summarization (Zhang et al., 2020), and sentence unshufﬂing for discourse representations (Lee et al., 2020).”