Semi-supervised Multitask Learning for Sequence Labeling

By Marek Rei. 2017

253 citations recorded by the source corpus.

We propose a sequence labeling framework with a secondary training objective, learning to predict surrounding words for every word in the dataset. This language modeling objective incentivises the system to learn general-purpose patterns of semantic and syntactic composition, which are also useful for improving accuracy on different sequence labeling tasks. The architecture was evaluated on a range of datasets, covering the tasks of error detection in learner texts, named entity recognition, chunking and POS-tagging. The novel language modeling objective provided consistent performance improvements on every benchmark, without requiring any additional annotated or unannotated data.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “Recent work has jointly trained image classiﬁers with tasks like relative position and colorization (Doersch and Zisserman, 2017), sequence taggers with language modeling (Rei, 2017), and reinforcement learning agents with predicting changes in the environment (Jader-berg et al., 2017).”
Universal Language Model Fine-tuning for Text Classification (2018, methodology): “This is the approach taken by Rei (2017) and Liu et al. (2018) who add a language modeling objective to the model that is trained jointly with the main task model.”
An Overview of Multi-Task Learning in Deep Neural Networks (2017, methodology): “The language modelling objective as employed by [Cheng et al., 2015] and [Rei, 2017] fulﬁls this role.”
Fine-tuned Language Models for Text Classification (2018, methodology): “This is the approach taken by Rei (2017) and Liu et al. (2018) who add a language modeling objective to the model that is trained jointly with the main task model.”
Sentence-State LSTM for Text Representation (2018, neutral): “26%; Rei (2017) perform multi-task”
How to Fine-Tune BERT for Text Classification? (2019, neutral): “Rei (2017) and Liu et al. (2018) use this method to train the language model and the main task model jointly.”
Hello, It’s GPT-2 - How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems (2019, positive): “…over the word sequence S = { w 1 , ..., w | S | } : Transfer learning based on such LM pretraining combined with the Transformer decoder model (Vaswani et al., 2017) resulted in signiﬁcant progress across many downstream tasks (Rei, 2017; Howard and Ruder, 2018; Radford et al., 2018, 2019).”
Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling (2018, methodology): “These LMs automatically capture abundant linguistic information and patterns from large text corpora, and can be applied to facilitate a wide range of NLP applications (Rei, 2017; Liu et al., 2018; Peters et al., 2018).”
Multi-Task Learning of Pairwise Sequence Classification Tasks over Disparate Label Spaces (2018, background): “Note also that several researchers have proposed using auxiliary tasks that are unsupervised (Plank et al., 2016; Rei, 2017), which also leads to a form of semi-supervised models.”
Controllable Paraphrase Generation with a Syntactic Exemplar (2019, background): “There has been a great deal of work in applying multi-task learning to improve performance on NLP tasks (Plank et al., 2016; Rei, 2017; Augen-stein and Søgaard, 2017; Bollmann et al., 2018, inter alia ).”
Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction (2018, background): “Most of these models share word-level representations (Collobert and Weston, 2008; Klerke et al., 2016; Luan et al., 2016, 2017a; Rei, 2017), while Peng et al. (2017) uses high-order cross-task factors.”
A Survey on Deep Learning for Named Entity Recognition (2018, methodology): “Rei [120] proposed a sequence labeling framework with a secondary objective – learning to predict surrounding words for each word in the dataset.”
Encode, Tag, Realize: High-Precision Text Editing (2019, methodology): “Methods for the sub-problem of error detection are similar in spirit to sentence compression systems, in that they are implemented as word-based neural sequence labelers (Rei, 2017; Rei et al., 2017).”
Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices (2019, background): “There are often also performance gains seen from adding auxiliary tasks to improve representation learning objectives [8, 33].”
A Survey on Deep Semi-Supervised Learning (2021, methodology): “Sequence learning IMDB [98], Rotten Tomatoes [102], 20 Newsgroups [116], DBpedia [96], CoNLL 2003 NER task [117], CoNLL 2000 Chunking task [118], Twitter POS Dataset [119], [120], Universal Dependencies(UD) [121], Combinatory Categorial Grammar (CCG) supertagging [122] [123], [124], [125], [126]”
Structural Information Preserving for Graph-to-Text Generation (2020, methodology): “Autoencoding loss by input reconstruction was mainly adopted on sequence labeling tasks, such as named entity recognition (NER) (Rei, 2017; Liu et al., 2018a; Jia et al., 2019), simile detection (Liu et al., 2018b) and sentiment analysis (Rei and Søgaard, 2019).”
Detecting semantic anomalies (2019, methodology): “A recent example of an application of this idea is [39], where the task of sequence labelling is augmented with the auxiliary self-supervised task of predicting neighbouring words, with shared representation learning parameters.”