Multi-Task Cross-Lingual Sequence Tagging from Scratch

By Zhilin Yang, Ruslan Salakhutdinov, William W. Cohen. 2016

227 citations recorded by the source corpus.

We present a deep hierarchical recurrent neural network for sequence tagging. Given a sequence of words, our model employs deep gated recurrent units on both character and word levels to encode morphology and context information, and applies a conditional random field layer to predict the tags. Our model is task independent, language independent, and feature engineering free. We further extend our model to multi-task and cross-lingual joint training by sharing the architecture and parameters. Our model achieves state-of-the-art results in multiple languages on several benchmark tasks including POS tagging, chunking, and NER. We also demonstrate that multi-task and cross-lingual joint training can improve the performance in various cases.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Revisiting Semi-Supervised Learning with Graph Embeddings (2016, neutral): “For example, word embed-dings trained from a language model can be applied to part-of-speech tagging, chunking and named entity recognition (Collobert et al., 2011; Yang et al., 2016).”
Gated-Attention Readers for Text Comprehension (2016, background): “Such embeddings have been previously shown to be helpful for tasks like Named Entity Recognition (Yang et al., 2016) and dealing with OOV tokens at test time (Dhingra et al.”
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF (2016, methodology): “A similar model has also been applied to Spanish and Portuguese NER (dos Santos et al., 2015) Ling et al. (2015) and Yang et al. (2016) also used BSLTM to compose character embeddings to word’s representation, which is similar to Lample et al. (2016).”
Latent Multi-Task Architecture Learning (2017, background): “ain types of) closely related tasks [2, 20]. [22] apply a variation of hard parameter sharing to multi-domain multi-task sequence tagging with a shared CRF layer and domain-speciﬁc projection layers. [30] also use hard parameter sharing to jointly learn different sequence-tagging tasks (NER, POS tag”
Sluice networks: Learning what to share between loosely related tasks (2017, methodology): “[30] also use hard parameter sharing to jointly learn different sequence-tagging tasks (NER, POS tagging, Chunking) across languages.”
A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks (2018, background): “Several prior works (Yang, Salakhutdinov, and Cohen 2016; Bingel and Søgaard 2017) avoid the question of the linguistic hierarchies between NLP tasks.”
DeepStruct: Pretraining of Language Models for Structure Prediction (2022, background): “In the context of NLP, weight sharing has been adopted in (Collobert and Weston, 2008; Yang et al., 2016; Liu et al., 2020).”
A Survey on Deep Learning for Named Entity Recognition (2018, methodology): “[105] developed a model to handle both cross-lingual and multitask joint training in a unified manner.”
Fast and Accurate Entity Recognition with Iterated Dilated Convolutions (2017, methodology): “Yang et al. (2016) use GRU-CRFs with GRU-composed character embeddings of words to train a single network on many tasks and languages.”
GCDT: A Global Context Enhanced Deep Transition Architecture for Sequence Labeling (2019, methodology): “, 2016), GRU (Yang et al., 2016), CNN (dos Santos and Zadrozny, 2014; Xin et al.”
Multi-task Domain Adaptation for Sequence Tagging (2016, unknown): “…block for learning representations in NLP and have achieved impressive performance in problems such as sequence tagging (Lample et al., 2016; Yang et al., 2016; Ma and Hovy, 2016), relation classiﬁcation (Xu et al., 2015; Zhang et al., 2015), and syntactic parsing (Kiperwasser and Goldberg,…”
On the Origin of Deep Learning (2017, methodology): “2 The Future of RNNs RNNs have been improved in a variety of different ways, like assembling the pieces together with Conditional Random Field (Yang et al., 2016), and together with CNN components (Ma and Hovy, 2016).”
Hierarchical Contextualized Representation for Named Entity Recognition (2019, unknown): “he-art models. Later, character-level embeddings are concatenated to enhance the representation of rare and out-ofvocabulary words, these embeddings are generated with LSTM (Lample et al.,2016), GRU (Yang et al., 2016), CNN (Ma and Hovy,2016), and recently IntNet (Xin et al.,2018). (Tran et al.,2017”
Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe (2017, background): “…utilization of recurrent neural networks (Hochreiter and Schmidhuber, 1997; Cho et al., 2014), deep neural networks achieved state-of-the-art results in many NLP areas like POS tagging (Ling et al., 2015), named entity recognition (Yang et al., 2016) or machine translation (Vaswani et al., 2017).”
Deep Learning for Aspect-Based Sentiment Analysis: A Comparative Review (2019, unknown): “…Shen, Yun, Lipton, Kronrod, & Anandkumar, 2017; Strubell, Verga, Belanger, & McCallum, 2017; Yang, Salakhutdinov, & Cohen, 2016);, ‘semantic role labelling’ (Do, Bethard, & Moens, 2017; Marcheggiani, Frolov, & Titov, 2017) and ‘Parts-Of-Speech (POS) tagging’ (Ma et al, 2016; Yang et al., 2016).”