Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss

By Barbara Plank, Anders Søgaard, Yoav Goldberg. Annual Meeting of the Association for Computational Linguistics, 2016

423 citations recorded by the source corpus.

Bidirectional long short-term memory (bi-LSTM) networks have recently proven successful for various NLP sequence modeling tasks, but little is known about their reliance to input representations, target languages, data set size, and label noise. We address these issues and evaluate bi-LSTMs with word, character, and unicode byte embeddings for POS tagging. We compare bi-LSTMs to traditional POS taggers across languages and data sizes. We also present a novel bi-LSTM model, which combines the POS tagging loss function with an auxiliary loss function that accounts for rare words. The model obtains state-of-the-art performance across 22 languages, and works especially well for morphologically complex languages. Our analysis suggests that bi-LSTMs are less sensitive to training data size and label corruptions (at small noise levels) than previously assumed.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/acd87e4f672f0b92ea4164414c213560c23bee52

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Balancing Training for Multilingual Neural Machine Translation (2020, background): “, 2013), but has become particularly prominent in recent years due the ability of neural networks to easily perform multi-task learning (Dong et al., 2015; Plank et al., 2016; Johnson et al., 2016).”
Cross-lingual and cross-domain discourse segmentation of entire documents (2017, methodology): “We used a modified version of (Plank et al., 2016) fixing the random seed and using standard SGD.”
Learning What’s Easy: Fully Differentiable Neural Easy-First Taggers (2017, background): “…we also show the accuracies re-ported by Gillick et al. (2016) for their byte-to-span system (trained separately on each language) and by Plank et al. (2016) for their state-of-the-art multi-task BILSTM tagger (these results are not fully comparable though, due to different treebank…”
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, background): “Plank et al. (2016) find that task perfor-”
When does deep multi-task learning work for loosely related document classification tasks? (2018, methodology): “Hard parameter sharing was ﬁrst introduced by (Caruana, 1993) and used with success for different tasks, for example in (Collobert et al., 2011; Klerke, Goldberg, and Søgaard, 2016; Plank, Søgaard, and Goldberg, 2016).”
From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning (2016, background): “The top level encodes global aspects of the input sequence and is sensitive to visually salient elements of its meaning.”
Strong Baselines for Neural Semi-Supervised Learning under Domain Shift (2018, methodology): “Recent NLP conferences witnessed a “tsunami” of deep learning papers (Manning, 2015), followed by what we call a multi-task learning “wave”: MTL has been successfully applied to a wide range of NLP tasks (Cohn and Specia, 2013; Cheng et al., 2015; Luong et al., 2015; Plank et al., 2016; Fang and Coh”
Multi-Task Learning of Pairwise Sequence Classification Tasks over Disparate Label Spaces (2018, background): “Note also that several researchers have proposed using auxiliary tasks that are unsupervised (Plank et al., 2016; Rei, 2017), which also leads to a form of semi-supervised models.”
A survey on sentiment analysis methods, applications, and challenges (2022, methodology): “In the work of Han et al. (2021) Transformer network revolutionized the area of NLP and replaced the usage of LSTM and Bi-LSTM.”
Multi-Task Learning for Sequence Tagging: An Empirical Study (2018, methodology): “In contrast, recent work considers stacked architectures (mostly for sequence tagging) in which tasks can supervise at different layers of a network (Søgaard and Goldberg, 2016; Klerke et al., 2016; Plank et al., 2016; Alonso and Plank, 2017; Bingel and Søgaard, 2017; Hashimoto et al., 2017).”
From Characters to Words to in Between: Do We Capture Morphology? (2017, background): “…of characters (Ling et al., 2015; Kim et al., 2016), character n-grams (Sperr et al., 2013; Wieting et al., 2016; Bojanowski et al., 2016; Botha and Blun-som, 2014), bytes (Plank et al., 2016; Gillick et al., 2016), or combinations thereof (Santos and Zadrozny, 2014; Qiu et al., 2014).”
Semi-supervised Multitask Learning for Sequence Labeling (2017, background): “Plank et al. (2016) described a related architecture for POS-tagging, predicting the frequency of each word together with the part-of-speech, and showed that this can improve tagging accuracy on low-frequency words.”
Robust Multilingual Part-of-Speech Tagging via Adversarial Training (2017, methodology): “Poor tagging accuracy on rare/unseen words is one of the bottlenecks in current POS taggers (Man-ning, 2011; Plank et al., 2016).”
Sluice networks: Learning what to share between loosely related tasks (2017, methodology): “Model We use a state-of-the-art BiLSTM-based sequence labeling model [23] as the building block of our model.”
From POS tagging to dependency parsing for biomedical event extraction (2018, background): “show that character-level word embeddings are specically useful to capture rare or unseen words in biomedical text data. Character-level word embeddings are useful for morphologically rich languages [38,27], and although English is not morphologically rich, the biomedical domain contains a wide vari”
DyNet: The Dynamic Neural Network Toolkit (2017, positive): “com/clab/stack-lstm-ner), POS-tagging [49], semantic role labeling [60] (https://github.”
Learning to select data for transfer learning with Bayesian Optimization (2017, methodology): “…Perceptron (in-house implementation with commonly used features pertaining to tags, words, case, preﬁxes, as well as preﬁxes and sufﬁxes) trained for 5 iterations with a learning rate of 0.2; and a state-of-the-art Bi-LSTM tagger (Plank et al., 2016) with word and character embeddings as input.”
English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too (2020, background): “When there is enough data in multiple languages with consistent annotations, training multilingual models becomes an option for cross-lingual transfer (Plank et al., 2016; Cotterell et al., 2017; Zeman et al., 2017).”