UDALM: Unsupervised Domain Adaptation through Language Modeling

By Constantinos F. Karouzos, Georgios Paraskevopoulos, Alexandros Potamianos. 2021

64 citations recorded by the source corpus.

In this work we explore Unsupervised Domain Adaptation (UDA) of pretrained language models for downstream tasks. We introduce UDALM, a fine-tuning procedure, using a mixed classification and Masked Language Model loss, that can adapt to the target domain distribution in a robust and sample efficient manner. Our experiments show that performance of models trained with the mixed loss scales with the amount of available target data and the mixed loss can be effectively used as a stopping criterion during UDA training. Furthermore, we discuss the relationship between A-distance and the target error and explore some limitations of the Domain Adversarial Training approach. Our method is evaluated on twelve domain pairs of the Amazon Reviews Sentiment dataset, yielding 91.74% accuracy, which is an 1.11% absolute improvement over the state-of-the-art.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, methodology): “Efforts on the data side have focused on intermediate ﬁne-tuning either by using unlabeled target domain data (Karouzos et al., 2021; Gururangan et al., 2020) or via labeled data from other tasks (Phang et al., 2018; Agha-janyan et al., 2021; Vu et al., 2020).”
AdaPrompt: Adaptive Model Training for Prompt-based NLP (2022, background): “UDALM (Karouzos et al., 2021) first trains PLMs by masked language”