Semi-Supervised Sequence Modeling with Cross-View Training

By Kevin Clark, Minh-Thang Luong, Christopher D. Manning, Quoc V. Le. 2018

343 citations recorded by the source corpus.

Unsupervised representation learning algorithms such as word2vec and ELMo improve the accuracy of many supervised NLP models, mainly because they can take advantage of large amounts of unlabeled text. However, the supervised models only learn from task-specific labeled data during the main training phase. We therefore propose Cross-View Training (CVT), a semi-supervised learning algorithm that improves the representations of a Bi-LSTM sentence encoder using a mix of labeled and unlabeled data. On labeled examples, standard supervised learning is used. On unlabeled examples, CVT teaches auxiliary prediction modules that see restricted views of the input (e.g., only part of a sentence) to match the predictions of the full model seeing the whole input. Since the auxiliary modules and the full model share intermediate representations, this in turn improves the full model. Moreover, we show that CVT is particularly effective when combined with multi-task learning. We evaluate CVT on five sequence tagging tasks, machine translation, and dependency parsing, achieving state-of-the-art results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, methodology): “BERTLARGE outperforms the existing SOTA, Cross-View Training with multi-task learning (Clark et al., 2018), by +0.2 on CoNLL-2003 NER Test.”
Unsupervised Data Augmentation for Consistency Training (2019, background): “In a nutshell, consistency training methods simply regularize model predictions to be invariant to small noise applied to either input examples [6, 7, 8] or hidden states [2, 4].”
Self-Training With Noisy Student Improves ImageNet Classification (2019, background): “Apart from self-training, another important line of work in semi-supervised learning [9, 85] is based on consistency training [6, 4, 53, 36, 70, 45, 41, 51, 10, 12, 49, 2, 38, 72, 74, 5, 81].”
Revisiting Self-Training for Neural Sequence Generation (2019, background): “Recent work on sequence generation employs auxiliary decoders (Clark et al., 2018) when processing unlabeled data, overall showing rather limited gains.”
MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification (2020, background): “Semi-supervised learning has received much attention in the NLP community (Gururangan et al., 2019; Clark et al., 2018; Yang et al., 2015), as unlabeled data is often plentiful compared to labeled data.”
Noisy Channel Language Model Prompting for Few-Shot Text Classification (2021, background): “Prior work in few-shot learning has used different approaches, including semi-supervised learning with data augmentation or consistency training (Miyato et al., 2017; Clark et al., 2018; Xie et al., 2020; Chen et al., 2020) and meta learning (Finn et al.”
Leveraging Just a Few Keywords for Fine-Grained Aspect Detection Through Weakly Supervised Co-Training (2019, background): “Several subsequent works have relaxed this assumption and shown co-training to be effective in much more general settings (Balcan et al., 2005; Chen et al., 2011; Collins and Singer, 1999; Clark et al., 2018).”
An Overview of Deep Semi-Supervised Learning (2020, background): “[25] proposed Cross-View Training, where the model is trained to produce consistent predictions across different views of the inputs.”
A Survey of Deep Learning for Scientific Discovery (2020, background): “This post https://ruder . io/semi-supervised/ gives a nice overview of diﬀerent co-training schemes, and [34, 179, 74] are some recent papers implementing this in text and images.”
Natural Language Processing Advancements By Deep Learning: A Survey (2020, unknown): “Semi-supervised Sequence Modeling [111] 92.61 Google BERT [112] 92.8 Contextual String Embeddings [79] 93.09 2) Relation Extraction: Relation Extraction aims to ﬁnd the semantic relationships between entity pairs.”
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, background): “Bhattacharjee et al. (2020) found that crossview training (Clark et al., 2018) leverages large amounts of unlabeled data better for task-specific applications in contrast to the general representations learned by BERT.”
The Cascade Transformer: an Application for Efficient Answer Sentence Selection (2020, background): “, 2015), masking part of the label (Clark et al., 2018), or use multiple modules for the teacher (Zhou and Li, 2005; Ruder and Plank, 2018).”
Distilling the Knowledge of BERT for Text Generation (2019, methodology): “Our best model outperforms Seq2Seq-OT (Chen et al., 2019) that utilizes optimal transport for sequence-level training, as well as the ELMo and CVT results reported in Clark et al. (2018).”
Prompt Consistency for Zero-Shot Task Generalization (2022, methodology): “, 2015), or previous consistency training that a single teacher distills to several students (Clark et al., 2018; Xie et al., 2020a), we perform distillation among a swarm of prompts where each prompt is a teacher and student at the same time, thus we term our method as swarm distillation.”