Neural Unsupervised Domain Adaptation in NLP—A Survey

By Alan Ramponi, Barbara Plank. 2020

295 citations recorded by the source corpus.

Deep neural networks excel at learning from labeled data and achieve state-of-the-art results on a wide array of Natural Language Processing tasks. In contrast, learning from unlabeled data, especially under domain shift, remains a challenge. Motivated by the latest advances, in this survey we review neural unsupervised domain adaptation techniques which do not require labeled target domain data. This is a more challenging yet a more widely applicable setup. We outline methods, from early traditional non-neural methods to pre-trained model transfer. We also revisit the notion of domain, and we uncover a bias in the type of Natural Language Processing tasks which received most attention. Lastly, we outline future directions, particularly the broader need for out-of-distribution generalization of future NLP.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

DEMix Layers: Disentangling Domains for Modular Language Modeling (2021, background): “Previous studies have shown the importance considering domains when adapting LMs (Ramponi and Plank, 2020; Gururangan et al., 2020).”
PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains (2021, background): “DA is a particular case of transfer learning, namely transductive transfer learning (Ramponi and Plank, 2020), in which T S and T T , the source and target tasks, are the same.”
Revisiting Transformer-based Models for Long Document Classification (2022, methodology): “…1.351 1.079 9.6 A comparison between TAPT and public available RoBERTa by (Lewis et al., 2020) We compare our TAPT-RoBERTa against publicly available domain-speciﬁc RoBERTa (Lewis et al., 2020), which are trained from scratch on biomedical articles and clinical notes, in hierarchical models.”
UDALM: Unsupervised Domain Adaptation through Language Modeling (2021, methodology): “Domain adversarial training is a dominant approach for UDA (Ramponi and Plank, 2020), in-1 spired by the theory for learning from different domains introduced in Ben-David et al. (2007, 2010).”
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, unknown): “, 2019) and unsupervised domain adaptation (Ramponi and Plank, 2020).”
Deep Unsupervised Domain Adaptation: A Review of Recent Advances and Perspectives (2022, methodology): “In recent NLP studies, it is a standard training strategy to fine-tune a transformer-based model with a small amount of data for a target application.”