Semi-supervised sequence tagging with bidirectional language models

By Matthew E. Peters, Bridger Waleed Ammar, Chandra Bhagavatula, Russell Power. 2017

669 citations recorded by the source corpus.

Pre-trained word embeddings learned from unlabeled text have become a standard component of neural network architectures for NLP tasks. However, in most cases, the recurrent network that operates on word-level representations to produce context sensitive representations is trained on relatively little labeled data. In this paper, we demonstrate a general semi-supervised approach for adding pretrained context embeddings from bidirectional language models to NLP systems and apply it to sequence labeling tasks. We evaluate our model on two standard datasets for named entity recognition (NER) and chunking, and in both cases achieve state of the art results, surpassing previous systems that use other forms of transfer or joint learning with additional labeled data and task specific gazetteers.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, methodology): “ELMo (Peters et al., 2017) generalizes traditional word embedding research along a different dimension.”
Dissecting Contextual Word Embeddings: Architecture and Representation (2018, background): “, 2018), named entity recognition (Peters et al., 2017), and text classification (Howard and Ruder, 2018).”
Deep Contextualized Word Representations (2018, methodology): “Following recent state-of-the-art systems (Lample et al., 2016; Peters et al., 2017), the baseline model uses pre-trained word embeddings, a”
Universal Language Model Fine-tuning for Text Classification (2018, methodology): “Recent approaches that concatenate embeddings derived from other tasks with the input at different layers (Peters et al., 2017; McCann et al., 2017; Peters et al., 2018) still train the main task model from scratch and treat pretrained embeddings as ﬁxed parameters, limiting their usefulness.”
Fine-tuned Language Models for Text Classification (2018, methodology): “Recent approaches concatenate embeddings derived from other tasks such as language modeling or machine translation with the input at different layers (Peters et al., 2017; McCann et al., 2017; Anonymous, 2018).”
Passage Re-ranking with BERT (2019, methodology): “Recently, neural models pretrained on a language modeling task, such as ELMo (Peters et al., 2017), OpenAI GPT (Radford et al.”
AllenNLP: A Deep Semantic Natural Language Processing Platform (2018, methodology): “Simply by changing a conﬁguration ﬁle, an experimenter can choose between pre-trained word embeddings, word embeddings concatenated with a character-level CNN encoding, or even pre-trained model token-in-context embeddings (Peters et al., 2017), which allows for easy controlled experimentation.”
Bottom-Up Abstractive Summarization (2018, positive): “Recent results have shown that better word representations can lead to significantly improved performance in sequence tagging tasks (Peters et al., 2017).”
ERNIE: Enhanced Language Representation with Informative Entities (2019, methodology): “Pre-trained language representation models, including feature-based (Mikolov et al., 2013; Pen-nington et al., 2014; Peters et al., 2017, 2018) and ﬁne-tuning (Dai and Le, 2015; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2018) approaches, can capture rich language information from…”
Learning protein sequence embeddings using information from structure (2019, background): “This is inspired partly by the recent success of unsupervised contextual embedding models using bidirectional recurrent neural network language models [15, 16] where word embeddings, learned as a function of their context, have been successfully transferred to other tasks.”
Sentence-State LSTM for Text Representation (2018, background): “In the second section of learning using additional language model objectives, obtaining an F-score of 86.26%; Peters et al. (2017) leverage character-level language models, obtaining an F-score of 91.93%, which is the current best result on the dataset.”
Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation (2018, methodology): “Representations obtained from an LM have been shown to obtain effective contextualized word representations (Peters et al. 2017; 2018).”
KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation (2019, background): “Recent works (Zhang et al., 2019; Peters et al., 2017; Liu et al., 2019a) utilize entity embeddings of large-scale knowledge bases to provide exter-nal knowledge for PLMs and improve their performance on various NLP tasks.”
Multi-Stage Document Ranking with BERT (2019, background): “Neural models pre-trained on language modeling tasks such as ELMo (Peters et al., 2017), Open-AI GPT (Radford et al., 2018), and BERT (De-vlin et al., 2019) have achieved impressive results on NLP tasks ranging from natural language inference to question answering.”
An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models (2019, background): “…information and high-level features of language, modeling syntax and semantics, producing state-of-the-art results across a wide range of tasks, such as named entity recognition (Peters et al., 2017), machine translation (Ramachandran et al., 2017) and text classiﬁ-cation (Howard and Ruder, 2018).”
Contextualized Word Representations for Reading Comprehension (2017, methodology): “Similar to our work, Peters et al. (2017) utilize the same pre-trained LM from J´ozefowicz et al. (2016) for sequence tagging tasks, keeping encoder weights ﬁxed during training.”