Empower Sequence Labeling with Task-Aware Neural Language Model

By Liyuan Liu, Jingbo Shang, Frank F. Xu, Xiang Ren, Huan Gui, Jian Peng, Jiawei Han. 2017

354 citations recorded by the source corpus.

Linguistic sequence labeling is a general approach encompassing a variety of problems, such as part-of-speech tagging and named entity recognition. Recent advances in neural networks (NNs) make it possible to build reliable models without handcrafted features. However, in many cases, it is hard to obtain sufficient annotations to train these models. In this study, we develop a neural framework to extract knowledge from raw texts and empower the sequence labeling task. Besides word-level knowledge contained in pre-trained word embeddings, character-aware neural language models are incorporated to extract character-level knowledge. Transfer learning techniques are further adopted to mediate different components and guide the language model towards the key knowledge. Comparing to previous methods, these task-specific knowledge allows us to adopt a more concise model and conduct more efficient training. Different from most transfer learning methods, the proposed framework does not rely on any additional supervision. It extracts knowledge from self-contained order information of training sequences. Extensive experiments on benchmark datasets demonstrate the effectiveness of leveraging character-level knowledge and the efficiency of co-training. For example, on the CoNLL03 NER task, model training completes in about 6 hours on a single GPU, reaching F_1 score of 91.71+/-0.10 without using any extra annotations.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, neutral): “(Chiu and Nichols, 2016) 91.62 ± 0.33 86.28 ± 0.26 ID-CNN-CRF (Strubell et al., 2017) 90.65 ± 0.15 86.84 ± 0.19 Tri-Trained LSTM (Lewis et al., 2016) 94.7 Shortcut LSTM (Wu et al., 2017) 95.08 97.53 JMT* (Hashimoto et al., 2017) 95.77 97.55 94.67 92.90 LM-LSTM-CNN-CRF (Liu et al., 2017) 95.96 ± 0.08”
Universal Language Model Fine-tuning for Text Classification (2018, methodology): “This is the approach taken by Rei (2017) and Liu et al. (2018) who add a language modeling objective to the model that is trained jointly with the main task model.”
Fine-tuned Language Models for Text Classification (2018, methodology): “This is the approach taken by Rei (2017) and Liu et al. (2018) who add a language modeling objective to the model that is trained jointly with the main task model.”
How to Fine-Tune BERT for Text Classification? (2019, neutral): “Rei (2017) and Liu et al. (2018) use this method to train the language model and the main task model jointly.”
Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling (2018, methodology): “These LMs automatically capture abundant linguistic information and patterns from large text corpora, and can be applied to facilitate a wide range of NLP applications (Rei, 2017; Liu et al., 2018; Peters et al., 2018).”
Structural Information Preserving for Graph-to-Text Generation (2020, methodology): “Autoencoding loss by input reconstruction was mainly adopted on sequence labeling tasks, such as named entity recognition (NER) (Rei, 2017; Liu et al., 2018a; Jia et al., 2019), simile detection (Liu et al., 2018b) and sentiment analysis (Rei and Søgaard, 2019).”
Chinese NER Using Lattice LSTM (2018, methodology): “It is similar with the structure of Liu et al. (2018) but not uses the highway layer.”
Simplify the Usage of Lexicon in Chinese NER (2019, methodology): “1The source code of this paper is publicly available at https://github.com/v-mipeng/ LexiconAugmentedNER. labeling problem, and the state-of-the-art results have been achieved using neural-network-based models (Huang et al., 2015; Chiu and Nichols, 2016; Liu et al., 2018).”
Natural Language Processing Advancements By Deep Learning: A Survey (2020, neutral): “Character-aware neural language models [74] 97.”
On the Transformer Growth for Progressive BERT Training (2020, background): “Thanks to the rapid increase of computing power, large-scale pre-training has been breaking the glass ceiling for natural language processing tasks (Liu et al., 2018; Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020).”
AlpacaTag: An Active Learning-based Crowd Annotation Framework for Sequence Tagging (2019, methodology): “We use the state-of-the-art sequence tagging model as our back-end model (Lample et al., 2016; Lin et al., 2017; Liu et al., 2018), which is based on bidirectional LSTM networks with a CRF layer (BLSTM-CRF).”
Low Resource Multi-Task Sequence Tagging - Revisiting Dynamic Conditional Random Fields (2020, methodology): “These approaches have been enhanced by leveraging pretrained language models (Peters et al., 2017; Liu et al., 2018) or using contextual embedding representations (Akbik et al., 2018) such as ELMO (Peters et al., 2018) or BERT (Devlin et al., 2019).”
Multi-Task Learning for Sequence Tagging: An Empirical Study (2018, background): “Another line of work considers language modeling as the auxiliary objective (Godwin et al., 2016; Rei, 2017; Liu et al., 2018).”
Improving Human Text Comprehension through Semi-Markov CRF-based Neural Section Title Generation (2019, methodology): “Similar to this work, it has also been shown that coupling an LM with an SCRF can improve segmentation through multi-task training (Lu et al., 2017; Liu et al., 2017).”
GCDT: A Global Context Enhanced Deep Transition Architecture for Sequence Labeling (2019, methodology): “Especially, Rei (2017) and Liu et al. (2017) build task-speciﬁc language models only on supervised data.”
Exploiting BERT for End-to-End Aspect-based Sentiment Analysis (2019, background): “, 2019), and competitive LSTM-CRF sequence labeling models (Lample et al., 2016; Ma and Hovy, 2016; Liu et al., 2018).”
TENER: Adapting Transformer Encoder for Named Entity Recognition (2019, neutral): “LM-LSTM-CRF (Liu et al., 2018) 91.”
Contextual String Embeddings for Sequence Labeling (2018, methodology): “Two prior works have suggested related approaches: The ﬁrst is Liu et al. (2017) that jointly train a character-level language model together with the sequence labeling BiLSTM.”
A Survey on Deep Learning for Named Entity Recognition (2018, methodology): “Such language-model-augmented knowledge has been empirically verified to be helpful in numerous sequence labeling tasks [19], [102], [120]–[122].”