Design Challenges and Misconceptions in Named Entity Recognition

By Lev-Arie Ratinov, Dan Roth. 2009

1,631 citations recorded by the source corpus.

We analyze some of the fundamental design challenges and misconceptions that underlie the development of an efficient and robust NER system. In particular, we address issues such as the representation of text chunks, the inference approach needed to combine local NER decisions, the sources of prior knowledge and how to use them within an NER system. In the process of comparing several solutions to these challenges we reach some surprising conclusions, as well as develop an NER system that achieves 90.8 F1 score on the CoNLL-2003 NER shared task, the best reported result for this dataset.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

Citations in the corpus

Semi-supervised sequence tagging with bidirectional language models (2017, methodology): “In both cases, we use the BIOES labeling scheme for the output tags, following previous work which showed it out-performs other options (e.g., Ratinov and Roth, 2009).”
Natural Language Processing (Almost) from Scratch (2011, methodology): “In that respect, these convex experiments show the performance of our word embeddings in a classical NLP framework. ar Xi v Following the Ratinov and Roth (2009) and Koo et al. (2008) setups, we generated 1, 000 Brown clusters using the implementation17 from Liang (2005).”
Sentence-State LSTM for Text Representation (2018, methodology): “For NER, we follow the standard split, and use the BIOES tagging scheme (Ratinov and Roth, 2009).”
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF (2016, background): “Most traditional high performance sequence labeling models are linear statistical models, including Hidden Markov Models (HMM) and Conditional Random Fields (CRF) (Ratinov and Roth, 2009; Passos et al., 2014; Luo et al., 2015), which rely heavily on hand-crafted features and task-speciﬁc resources.”
Named Entity Recognition with Bidirectional LSTM-CNNs (2015, methodology): “rson 1,074,363 Total 2,344,578 Table 1: Number of entries for each category in the lexicon 2.1.3 Lexicons Almost all state of the art NER systems make use of lexicons as a form of external knowledge (Ratinov and Roth, 2009; Passos et al., 2014; Durrett and Klein, 2014; Luo et al., 2015). For each of”
End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures (2016, methodology): “Traditional systems treat this task as a pipeline of two separated tasks, i.e., named entity recognition (NER) (Nadeau and Sekine, 2007; Ratinov and Roth, 2009) and relation extraction (Zelenko et al., 2003; Zhou et al., 2005), but recent studies show that end-to-end (joint) modeling of entity and…”
Lexicon Infused Phrase Embeddings for Named Entity Resolution (2014, methodology): “This process produces a hierarchical clustering of the word types in the corpus, and these clusterings have been found useful in many applications (Ratinov and Roth, 2009; Koo et al., 2008; Miller et al., 2004).”
Question Answering as Global Reasoning Over Semantic Abstractions (2018, methodology): “• Span which can contains labels for spans of text; we instantiated Shallow-Parse (Punyakanok and Roth 2001), Quantities (Roy, Vieira, and Roth 2015), NER (Ratinov and Roth 2009; Redman, Sammons, and Roth 2016)).”
Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling (2018, methodology): “In both cases, we use the BIOES labeling scheme (Ratinov and Roth, 2009) and use the micro-averaged F1 as the evaluation metric.”
Learning multilingual named entity recognition from Wikipedia (2013, background): “Other semi-supervised approaches improve performance by incorporating knowledge from unlabelled text in a supervised ner system, through: highly-predictive features from related tasks [4]; selected output of a supervised system [86,87, 37]; jointly modelling labelled and unlabelled [74] or partially”
FedNER: Medical Named Entity Recognition with Federated Learning (2020, background): “Sufﬁcient labeled data is critical for these meth-ods to train accurate medical NER model (Ratinov and Roth, 2009).”
Removing the Training Wheels: A Coreference Dataset that Entertains Humans and Challenges Computers (2015, methodology): “A system by Ratinov and Roth (2012) demonstrates good performance by using Wikipedia knowledge to strengthen a multi-pass rule based system.”
Re-ranking for joint named-entity recognition and linking (2013, methodology): “Nearly all EL systems therefore use a pipeline architecture: they run a Named-Entity Recognition (NER) [6, 17] system on the text to identify the boundaries of names.”
Word Embeddings through Hellinger PCA (2013, methodology): “Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 482–490, Gothenburg, Sweden, April 26-30 2014. c©2014 Association for Computational Linguistics”
Knowledge Graphs (2020, neutral): “The NER task identifies mentions of named entities in a text [368, 433], typically targetting mentions of people, organisations, locations, and potentially other types [319, 369, 572].”
Joint Entity Recognition and Disambiguation (2015, methodology): “We take three state-of-art NER systems: NereL (Sil and Yates, 2013), UIUC NER (Ratinov and Roth, 2009) and Stanford NER (Finkel et al., 2005 For entity linking systems, NereL, Kul09 (Kulkarni et al., 2009) and Hof11 (Hoffart et al., 2011) are compared with our models.”
Unsupervised biomedical named entity recognition: Experiments with clinical and biological texts (2013, background): “Such state-of-the-art NER systems have achieved performance as high as human annotators [10,11].”
Multi-Task Cross-Lingual Sequence Tagging from Scratch (2016, methodology): “We also use the BIOES (Begin, Inside, Outside, End, Single) tagging scheme (Ratinov and Roth, 2009).”
Learning Structured Perceptrons for Coreference Resolution with Latent Antecedents and Non-local Features (2014, background): “…(Collins and Roark, 2004; Huang, 2008; Zhang and Clark, 2008) and linearization (Bohnet et al., 2011), and even simpler structured prediction problems, where early updates are not even necessary, such as part-of-speech tagging (Collins, 2002) and named entity recognition (Ratinov and Roth, 2009).”