Fast and Accurate Entity Recognition with Iterated Dilated Convolutions

By Emma Strubell, Pat Verga, David Belanger, Andrew McCallum. 2017

430 citations recorded by the source corpus.

Today when many practitioners run basic NLP on the entire web and large-volume traffic, faster methods are paramount to saving time and energy costs. Recent advances in GPU hardware have led to the emergence of bi-directional LSTMs as a standard method for obtaining per-token vector representations serving as input to labeling tasks such as NER (often followed by prediction in a linear-chain CRF). Though expressive and accurate, these models fail to fully exploit GPU parallelism, limiting their computational efficiency. This paper proposes a faster alternative to Bi-LSTMs for NER: Iterated Dilated Convolutional Neural Networks (ID-CNNs), which have better capacity than traditional CNNs for large context and structured prediction. Unlike LSTMs whose sequential processing on sentences of length N requires O(N) time even in the face of parallelism, ID-CNNs permit fixed-depth convolutions to run in parallel across entire documents. We describe a distinct combination of network structure, parameter sharing and training procedures that enable dramatic 14-20x test-time speedups while retaining accuracy comparable to the Bi-LSTM-CRF. Moreover, ID-CNNs trained to aggregate context from the entire document are more accurate than Bi-LSTM-CRFs while attaining 8x faster test time speeds.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks (2018, methodology): “tting Datasets and evaluation metrics We use labeled data from different sources to train and evaluate our model. For NER, we use the English portion of OntoNotes 5.0 (Pradhan et al. 2013). Following Strubell et al. (2017), we use the same data split as used for coreference resolution in the CoNLL-2”
Double Embeddings and CNN-based Sequence Labeling for Aspect Extraction (2018, methodology): “, 1995; Kim, 2014) is recently adopted for named entity recognition (Strubell et al., 2017).”
Conditional probing: measuring usable information beyond a baseline (2021, methodology): “For all tasks except sentiment, we use the Ontonotes v5 corpus (Weischedel et al., 2013), recreating the splits used in the CoNLL 2012 shared task, as veriﬁed against the split statistics provided by Strubell et al. (2017).”
Template-Based Named Entity Recognition Using BART (2021, methodology): “…BiLSTM (Hochreiter and Schmidhuber, 1997) and BERT (Devlin et al., 2019) is used to represent the input text, and softmax (Chiu and Nichols, 2016; Strubell et al., 2017; Cui and Zhang, 2019) or CRF (Lample et al., 2016; Ma and Hovy, 2016; Luo et al., 2020) output layers to assign named entity…”
A Survey on Deep Learning for Named Entity Recognition (2018, methodology): “Neurons contributing to a single highlighted neuron in the last layer are highlighted [89].”
Contextual Memory Trees (2018, background): “There are works on leveraging memory systems to perform few-shot learning tasks ((Snell et al., 2017; Strubell et al., 2017; Santoro et al., 2016)).”
Marginal Likelihood Training of BiLSTM-CRF for Biomedical Named Entity Recognition from Disjoint Label Sets (2018, background): “More recently, neural network based approaches have become state-of-the-art (Lample et al., 2016; Strubell et al., 2017; Peters et al., 2017).”
Deep Active Learning for Named Entity Recognition (2017, background): “However, CNN word-level encoders are considerably faster (Strubell et al., 2017), which is crucial for the iterative retraining in our active learning scheme.”
TENER: Adapting Transformer Encoder for Named Entity Recognition (2019, methodology): “Therefore, CNN has been proposed by (Strubell et al., 2017; Gui et al., 2019a) to encode words concurrently.”
Unified Named Entity Recognition as Word-Word Relation Classification (2021, methodology): “Studies of NER have gradually evolved initially from the flat NER (Lample et al. 2016; Strubell et al. 2017), late to the overlapped NER (Yu et al.”
A Comprehensive Survey on Automatic Knowledge Graph Construction (2023, background): “[101] improved upon the CNN with dilated convolutions that enlarge the perception field by omitting some of the input to enhance generalization.”
Named Entity Recognition as Dependency Parsing (2020, neutral): “6 Results on Flat NER We evaluate our system on ﬁve corpora for ﬂat NER ( CONLL 2002 (Dutch, Spanish) 3), e.g. the system of Chiu and Nichols (2016); Strubell et al. (2017).”
A Unified MRC Framework for Named Entity Recognition (2019, methodology): “To separate the influence from large-scale BERT pretraining, we compare the LSTM-CRF tagging model (Strubell et al., 2017) with other MRC based models such as QAnet (Yu et al.”
Simplify the Usage of Lexicon in Chinese NER (2019, background): “Another popular architecture for sequence modeling is the convolution network (Kim, 2014), which has been proved (Strubell et al., 2017) to be effective for Chinese NER.”