Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks

By Zhilin Yang, Ruslan Salakhutdinov, William W. Cohen. 2016

356 citations recorded by the source corpus.

Recent papers have shown that neural networks obtain state-of-the-art performance on several different sequence tagging tasks. One appealing property of such systems is their generality, as excellent performance can be achieved with a unified architecture and without task-specific feature engineering. However, it is unclear if such systems can be used for tasks without large amounts of training data. In this paper we explore the problem of transfer learning for neural sequence taggers, where a source task with plentiful annotations (e.g., POS tagging on Penn Treebank) is used to improve performance on a target task with fewer available annotations (e.g., POS tagging for microblogs). We examine the effects of transfer learning for deep hierarchical recurrent networks across domains, applications, and languages, and show that significant improvement can often be obtained. These improvements lead to improvements over the current state-of-the-art on several well-studied tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Semi-supervised sequence tagging with bidirectional language models (2017, methodology): “Our baseline sequence tagging model is a hierarchical neural tagging model, closely following a number of recent studies (Ma and Hovy, 2016; Lample et al., 2016; Yang et al., 2017; Chiu and Nichols, 2016) (left side of Figure 2).”
Sentence-State LSTM for Text Representation (2018, methodology): “ byMa and Hovy(2016) andLample et al.(2016), who also use bidirectional RNNCRF structures. In contrast, S-LSTM gives the best reported results under the same settings. In the second section of Table7,Yang et al. (2017) use cross-domain data, obtaining an Fscore of 91.26%;Rei(2017) perform multi-task”
GCDT: A Global Context Enhanced Deep Transition Architecture for Sequence Labeling (2019, methodology): “More notably, our GCDT surpasses the models that exploit additional task-speciﬁc resources or annotated corpora (Luo et al., 2015; Yang et al., 2017b; Chiu and Nichols, 2016).”
Teaching a New Dog Old Tricks: Resurrecting Multilingual Retrieval Using Zero-Shot Learning (2019, background): “In the past few years, transfer learning between languages has been proven to be a remarkably effective approach for low-resource multilingual tasks (e.g. [16, 17, 29, 38] ).”
Natural Language Processing Advancements By Deep Learning: A Survey (2020, unknown): “Character-aware neural language models [74] 97.53 Transfer Learning + GRU [75] 97.55 Bi-directional LSTM + CNNs + CRF [76] 97.55 Adversarial Training + Bi-LSTM [77] 97.59 Character Composition + Bi-LSTM [78] 97.78 String Embedding + LSTM [79] 97.85 Meta-BiLSTM [80] 97.96”
Empower Sequence Labeling with Task-Aware Neural Language Model (2017, background): “E.g., although a naive co-training setting could be effective to directly use the output from character-level layers in several scenarios (Yang, Salakhutdinov, and Cohen 2017), for our two tasks, it would hurt the performance.”
A Survey on Recent Advances in Named Entity Recognition from Deep Learning models (2018, unknown): “Yang et al. (2017) Kim Sang, 2002; Cucerzan and Yarowsky, 2002) and 2003 (Tjong Kim Sang and De Meulder, 2003) and for biomedical NER (Saha et al., 2009), but had not been used in neural NER systems.”
AdaRNN: Adaptive Learning and Forecasting of Time Series (2021, methodology): “Several methods applied pre-training to RNNs [4, 8, 11, 15, 51] which is similar to CNN-based pre-training.”
Chinese NER Using Lattice LSTM (2018, methodology): “Character bigrams have been shown useful for representing characters in word segmentation (Chen et al., 2015; Yang et al., 2017a).”
Cross-Lingual Transfer Learning for Question Answering (2019, background): “It is also widely studied on NLP tasks such as sequence tagging (Yang et al., 2017), syntactic parsing (McClosky et al., 2010), and named entity recognition (Chiticariu et al., 2010).”
Efficient Large-Scale Domain Classification with Personalized Attention (2018, background): “…(Zhang and Wang, 2016; Liu and Lane, 2016; Kim et al., 2017b), transfer learning (El-Kahky et al., 2014; Kim et al., 2015a,c; Chen et al., 2016a; Yang et al., 2017), domain adaptation (Kim et al., 2016; Jaech et al., 2016; Liu and Lane, 2017; Kim et al., 2017d,c) and contextual signals…”
CoNet: Collaborative Cross Networks for Cross-Domain Recommendation (2018, unknown): “processing [5,43] and computer vision [44,27,8].”
Hierarchical Contextualized Representation for Named Entity Recognition (2019, methodology): “(Zhang, Liu, and Song 2018) simultaneously model the sub-states for individual words and an overall sentence-level state.”
Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling (2019, methodology): “RNN-based models for sequence labeling can be adapted across domains by manipulating the input or output layers individually (e.g., Yang et al., 2016) or simultaneously (Lin and Lu, 2018).”
A Survey on Deep Learning for Named Entity Recognition (2018, background): “[153] first investigated the transferability of different layers of representations.”