Unsupervised Domain Clusters in Pretrained Language Models

By Roee Aharoni, Yoav Goldberg. 2020

282 citations recorded by the source corpus.

The notion of “in-domain data” in NLP is often over-simplistic and vague, as textual data varies in many nuanced linguistic aspects such as topic, style or level of formality. In addition, domain labels are many times unavailable, making it challenging to build domain-specific systems. We show that massive pre-trained language models implicitly learn sentence representations that cluster by domains without supervision – suggesting a simple data-driven definition of domains in textual data. We harness this property and propose domain data selection methods based on such models, which require only a small set of in-domain monolingual data. We evaluate our data selection methods for neural machine translation across five diverse domains, where they outperform an established approach as measured by both BLEU and precision and recall with respect to an oracle selection.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

DEMix Layers: Disentangling Domains for Modular Language Modeling (2021, background): “Dense training leaves variation in the data to be implicitly discovered (Aharoni and Goldberg, 2020), assuming that models will be able to fit all domains equally well.”
AdaPrompt: Adaptive Model Training for Prompt-based NLP (2022, background): “The selection model can be a supervised classifier (Aharoni and Goldberg, 2020), similarity-based metric (Plank and Van Noord, 2011) or language model perplexity (Moore and Lewis, 2010).”
Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment (2023, background): “AdapterSoup [22] is developed to address cross-domain task adaptation, which first trains multiple adapters based on various domains and then employs domain clustering [81] to select the most appropriate top-k adapters for the new domain.”
DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference (2021, unknown): “Indeed, the fact that pre-trained transformers can be used to create meaningful clusters has been shown in other recent works (c.f. Aharoni and Goldberg (2020); Joshi et al. (2020)).”
Train No Evil: Selective Masking for Task-guided Pre-training (2020, methodology): “In this way, PLMs can utilize more data to better learn domain-specific language patterns (Alsentzer et al., 2019; Lee et al., 2019; Sung et al., 2019; Xu et al., 2019; Aharoni and Goldberg, 2020).”
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis (2020, methodology): “ile GloVe features are 300 dimensional token embeddings, for BERT, we utilize the BERT-base-uncased pre-trained model. This model comprises of 12 stacked Transformer layers. Aligned with recent works [1], we choose the utterance vector ul to be the average representation of the tokens from the final”
DirectProbe: Studying Representations without Classifiers (2021, background): “Grouping data points in this fashion is related to a well-studied problem, namely clustering, and several recent works have looked at clustering of contextualized representations (Reimers et al., 2019; Aharoni and Goldberg, 2020; Gupta et al., 2020).”
Neural Unsupervised Domain Adaptation in NLP—A Survey (2020, background): “Recent work investigates data representation and cosine similarity for MT data selection (Aharoni and Goldberg, 2020).”
Nearest Neighbor Machine Translation (2020, methodology): “We use the multi-domains dataset (Koehn & Knowles, 2017), re-split by Aharoni & Goldberg (2020) for the domain adaptation experiments.”
In-context Examples Selection for Machine Translation (2022, methodology): “For the out-of-domain evaluation, we use the multi-domain dataset from Aharoni and Goldberg (2020) for the following domains: Medical, Law, IT, and Koran.”
The Hitchhiker’s Guide to Computational Linguistics in Suicide Prevention (2021, background): “For example, a recent study used embedding models to map (cluster) textual data to distinct domains, thus allowing the researchers to determine the characteristics of the source of the data (e.g., medically oriented texts; Aharoni & Goldberg, 2020).”
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020, methodology): “Concurrent to our work, Aharoni and Goldberg (2020) propose data selection methods for NMT based on cosine similarity in embedding space, using DISTILBERT (Sanh et al.”
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (2022, unknown): “However, other methods for discovering domains, including unsupervised techniques (Aharoni and Goldberg, 2020; Chronopoulou et al., 2022), may yield better expert assignments.”
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, background): “Aharoni and Goldberg (2020) found domain-specific clusters in pre-trained language models and showed how these could be exploited for data selection in domain-sensitive training.”