Strong Baselines for Neural Semi-Supervised Learning under Domain Shift

By Sebastian Ruder, Barbara Plank. 2018

178 citations recorded by the source corpus.

Novel neural models have been proposed in recent years for learning under domain shift. Most models, however, only evaluate on a single task, on proprietary datasets, or compare to weak baselines, which makes comparison of models difficult. In this paper, we re-evaluate classic general-purpose bootstrapping approaches in the context of neural networks under domain shifts vs. recent neural approaches and propose a novel multi-task tri-training method that reduces the time and space complexity of classic tri-training. Extensive experiments on two benchmarks for part-of-speech tagging and sentiment analysis are negative: while our novel method establishes a new state-of-the-art for sentiment analysis, it does not fare consistently the best. More importantly, we arrive at the somewhat surprising conclusion that classic tri-training, with some additions, outperforms the state-of-the-art for NLP. Hence classic approaches constitute an important and strong baseline.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, background): “It is also possible to use multiple models or prediction modules for the teacher, such as in tri-training (Zhou and Li, 2005; Ruder and Plank, 2018).”
Do Not Have Enough Data? Deep Learning to the Rescue! (2019, background): “Text classiﬁcation is an instance of the supervised learning problem (Russell and Norvig 2016) over textual data.”
An Overview of Deep Semi-Supervised Learning (2020, methodology): “As for unsupervised domain adaptation, semi-supervised methods, such as consistency regularization [140, 93, 45], co-regularization [89] or proxy labeling [132, 131] demonstrated their effectiveness in domain adaptation.”
The Cascade Transformer: an Application for Efficient Answer Sentence Selection (2020, background): “…variants of ST have 2 Also known as auxiliary or diagnostic classiﬁers. been proposed, including treating teacher predictions as soft labels (Hinton et al., 2015), masking part of the label (Clark et al., 2018), or use multiple modules for the teacher (Zhou and Li, 2005; Ruder and Plank, 2018).”
Neural Unsupervised Domain Adaptation in NLP—A Survey (2020, methodology): “Data-centric methods focus on the data aspect and either involve pseudo-labeling (or bootstrapping) to bridge the domain gap (Abney, 2007; Zhu and Goldberg, 2009; Ruder and Plank, 2018; Cui and Bollegala, 2019), data selection (Axelrod et al.”
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis (2020, background): “The non-redundancy is achieved by enforcing a soft orthogonality constraint between the two representations [5, 25, 47].”
KinGDOM: Knowledge-Guided DOMain Adaptation for Sentiment Analysis (2020, methodology): “However, in DANN+, us-ing Adam optimizer leads to substantial jump in performance which can comfortably surpass many of the recent advanced domain adaptation methods – CMD (Zellinger et al., 2017), VFAE (Louizos et al., 2015), ASym (Saito et al., 2017) and MT-Tri (Ruder and Plank, 2018).”
UDALM: Unsupervised Domain Adaptation through Language Modeling (2021, background): “Ruder and Plank (2018) found that the domain adversarial loss did not help for their experiments on the Amazon reviews dataset.”