End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF

By Xuezhe Ma, Eduard H. Hovy. 2016

2,816 citations recorded by the source corpus.

State-of-the-art sequence labeling systems traditionally require large amounts of task-specific knowledge in the form of hand-crafted features and data pre-processing. In this paper, we introduce a novel neutral network architecture that benefits from both word- and character-level representations automatically, by using combination of bidirectional LSTM, CNN and CRF. Our system is truly end-to-end, requiring no feature engineering or data pre-processing, thus making it applicable to a wide range of sequence labeling tasks. We evaluate our system on two data sets for two sequence labeling tasks --- Penn Treebank WSJ corpus for part-of-speech (POS) tagging and CoNLL 2003 corpus for named entity recognition (NER). We obtain state-of-the-art performance on both the two data --- 97.55\% accuracy for POS tagging and 91.21\% F1 for NER.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “Our models use two layer CNN-BiLSTM encoders (Chiu and Nichols, 2016; Ma and Hovy, 2016; Lample et al., 2016) and task-speciﬁc prediction modules.”
Deep Contextualized Word Representations (2018, methodology): “Similar to WSD, the biLM representations are competitive with carefully tuned, task speciﬁc biLSTMs (Ling et al., 2015; Ma and Hovy, 2016).”
Semi-supervised sequence tagging with bidirectional language models (2017, methodology): “The baseline sequence tagger uses 30 dimensional character embeddings and a CNN with 30 ﬁlters of width 3 characters followed by a tanh non-linearity for the token character encoder.”
SciBERT: A Pretrained Language Model for Scientific Text (2019, methodology): “multilayer BiLSTM to token embeddings. For text classiﬁcation, we apply a multilayer perceptron on the ﬁrst and last BiLSTM states. For sequence tagging, weuse a CRF on top of the BiLSTM, as done in (Ma and Hovy, 2016). For dependency parsing we use the biafﬁne attention model from Dozat and Manning”
When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation? (2018, methodology): “Pre-trained word embeddings have proven to be highly useful in neural network models for NLP tasks such as sequence tagging (Lample et al., 2016; Ma and Hovy, 2016) and text classification (Kim, 2014).”
Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks (2017, methodology): “, 2015), the BiLSTM-CNN-CRF (Ma and Hovy, 2016) architecture and the BiLSTM-LSTM-CRF architecture (Lample et al.”
Sentence-State LSTM for Text Representation (2018, methodology): “Bi-directional extensions have become a standard conﬁguration for achieving state-of-the-art accuracies among various tasks (Wen et al., 2015; Ma and Hovy, 2016; Dozat and Manning, 2017).”
FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP (2019, background): “Many recently proposed approaches go beyond the initial “one word, one embedding” paradigm to better model additional features such as subword structures (Ma and Hovy, 2016; Bojanowski et al., 2017) and meaning ambiguity (Peters et al.”
Harnessing Deep Neural Networks with Logic Rules (2016, methodology): “Further integration of the list rule (Row 3) provides a second boost in performance, achieving an F1 score very close to the best-performing systems including Joint-NER-EL (Luo et al., 2015) (Row 8), a probabilistic graphical model optimizing NER and entity linking jointly with massive external reso”
Adversarial Dropout for Supervised and Semi-supervised Learning (2017, background): “Extensive empirical research proved the effectiveness of dropout techniques (Krizhevsky, Sutskever, and Hinton 2012; Ma and Hovy 2016), and dropout has recently been analyzed theoretically to reveal the rational of how and why dropout works (Srivastava et al.”
Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging (2017, methodology): “Further we will focus on Long-Short-Term-Memory (LSTM) Networks (Hochreiter and Schmidhuber, 1997b), as those demonstrated state-of-the-art performance for a wide variety of sequence tagging tasks (Ma and Hovy, 2016; Lample et al., 2016; Søgaard and Goldberg, 2016).”
Word Representation Models for Morphologically Rich Languages in Neural Machine Translation (2016, background): “Several authors have proposed convolutional neural networks over character sequences, as part of neural models for part of speech tagging (Santos and Zadrozny, 2014), named entity recognition (Ma and Hovy, 2016; Chiu and Nichols, 2015), language modelling (Kim et al.”
Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling (2018, background): “Many attempts have been made to conduct end-to-end learning and build reliable models without handcrafted features (Chiu and Nichols, 2016; Lample et al., 2016; Ma and Hovy, 2016).”
Few-Shot Named Entity Recognition: A Comprehensive Study (2020, neutral): “, 2016) such as Bidrectional LSTMs (Chiu and Nichols, 2016) and adding CRFs to capture structures (Ma and Hovy, 2016).”
Adaptive Methods for Nonconvex Optimization (2018, methodology): “We use the popular CNN-LSTM-CRF model [5], [19] for NER task.”