Bidirectional LSTM-CRF Models for Sequence Tagging

By Zhiheng Huang, Wei Xu, Kai Yu. 2015

4,469 citations recorded by the source corpus.

In this paper, we propose a variety of Long Short-Term Memory (LSTM) based models for sequence tagging. These models include LSTM networks, bidirectional LSTM (BI-LSTM) networks, LSTM with a Conditional Random Field (CRF) layer (LSTM-CRF) and bidirectional LSTM with a CRF layer (BI-LSTM-CRF). Our work is the first to apply a bidirectional LSTM CRF (denoted as BI-LSTM-CRF) model to NLP benchmark sequence tagging data sets. We show that the BI-LSTM-CRF model can efficiently use both past and future input features thanks to a bidirectional LSTM component. It can also use sentence level tag information thanks to a CRF layer. The BI-LSTM-CRF model can produce state of the art (or close to) accuracy on POS, chunking and NER data sets. In addition, it is robust and has less dependence on word embedding as compared to previous observations.

Subfields: Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Model Architectures

Citations in the corpus

Revisiting Semi-Supervised Learning with Graph Embeddings (2016, background): “For example, word embeddings trained from a language model can be applied to partof-speech tagging, chunking and named entity recognition (Collobert et al., 2011; Huang et al., 2015).”
Neural Architectures for Named Entity Recognition (2016, methodology): “More recently, Huang et al. (2015) presented a model similar to our LSTM-CRF, but using hand-crafted spelling features.”
Recent Trends in Deep Learning Based Natural Language Processing (2017, methodology): “[133] concatenated word embeddings and manually-designed wordlevel features and employed bidirectional LSTM to model arbitrarily long context.”
Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks (2017, methodology): “…gradient normalization threshold of 1 should be used, that variational dropout (Gal and Ghahramani, 2016) applied to the output units as well as the recurrent units of an LSTM-layer is optimal and that the CRF-classifier presented (Huang et al., 2015) should be preferred over a softmax classifier.”
Sentence-State LSTM for Text Representation (2018, methodology): “,hn for calculating the conditional probabilities of label sequences (Huang et al., 2015; Ma and Hovy, 2016):”
Recurrent Neural Networks for Classifying Relations in Clinical Notes (2017, neutral): “We also plan to experiment with more advanced architectures such as bidirectional LSTM [36] that can more efficiently use both previous context features and succeeding context features and model subtle differences such as passive voice vs.”
Towards Open Intent Discovery for Conversational Text (2019, methodology): “CRFs effectively utilize the correlations between labels in a sequence neighborhood to predict the best label sequence for a given input.”
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF (2016, methodology): “Huang et al. (2015) used BLSTM for word-level representations and CRF for jointly label decoding, which is similar to our model.”
Globally Normalized Transition-Based Neural Networks (2016, methodology): “Yao et al. (2014) and Zheng et al. (2015) introduce recurrence into the model and Huang et al. (2015) ﬁnally combine CRFs and LSTMs.”
Multilingual Language Processing From Bytes (2015, methodology): “Huang et al. (2015) improve on Collobert’s results by using a bidirectional LSTM with a CRF layer where the inputs are features describing the words in each sentence.”
Named Entity Recognition with Bidirectional LSTM-CNNs (2015, methodology): “( 0.44) 86.05 86.37 86.21 ( 0.20) BLSTM-CNN + emb + lex 90.41 91.15 90.77 ( 0.35) 85.97 86.83 86.39 ( 0.19) Collobert et al. (2011b) - - 88.67 - - - Collobert et al. (2011b) + lexicon - - 89.59 - - - Huang et al. (2015) - - 90.10 - - - Ratinov and Roth (2009)11 91.20 90.50 90.80 82.00 84.95 83.45 Li”
End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures (2016, background): “Recently, Huang et al. (2015) showed that building a conditional random ﬁeld (CRF) layer on top of bidirectional LSTM-RNNs performs comparably to the state-of-the-art methods in the part-of-speech (POS) tagging, chunking, and NER.”
Structured Prediction Energy Networks (2015, methodology): “It is natural to parametrize the potentials of a CRF using deep features (LeCun et al., 2006; Collobert et al., 2011; Jaderberg et al., 2014; Huang et al., 2015; Chen et al., 2014; Schwing & Urtasun, 2015; Chen et al., 2015).”
Harvesting Paragraph-level Question-Answer Pairs from Wikipedia (2018, methodology): “We frame the problem of identifying candidate answer spans from a paragraph as a sequence labeling task and base our model on the BiLSTM-CRF approach for named entity recognition (Huang et al., 2015).”
Recognition (2003, neutral): “To test this, we introduce an exploratory architecture that inserts a receptor-glomerular bottleneck into a standard BiLSTM-CRF model.”
Semi-supervised Multitask Learning for Sequence Labeling (2017, methodology): “(2001)), which allows the network to look for the most optimal path through all possible label sequences (Huang et al., 2015; Lample et al., 2016).”
Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging (2017, methodology): “We use a BiLSTM-network for sequence tagging as described in (Huang et al., 2015; Ma and Hovy, 2016; Lample et al., 2016).”
When is multitask learning effective? Semantic sequence prediction under varying data conditions (2016, methodology): “Bi-LSTM have recently successfully been used for a variety of tasks (Col-lobert et al., 2011; Huang et al., 2015; Dyer et al., 2015; Ballesteros et al., 2015; Kiperwasser and Goldberg, 2016; Liu et al., 2015; Plank et al., 2016).”