Learning to select data for transfer learning with Bayesian Optimization

By Sebastian Ruder, Barbara Plank. 2017

203 citations recorded by the source corpus.

Domain similarity measures can be used to gauge adaptability and select suitable data for transfer learning, but existing approaches define ad hoc measures that are deemed suitable for respective tasks. Inspired by work on curriculum learning, we propose to learn data selection measures using Bayesian Optimization and evaluate them across models, domains and tasks. Our learned measures outperform existing domain similarity measures significantly on three tasks: sentiment analysis, part-of-speech tagging, and parsing. We show the importance of complementing similarity with diversity, and that learned measures are–to some degree–transferable across models, domains, and even tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks (2019, methodology): “We calculate the Jensen-Shannon divergence based on term distributions (Ruder and Plank, 2017) between the domains used to train BERT (books and Wikipedia) and each MNLI domain.”
Toward Understanding Catastrophic Forgetting in Continual Learning (2019, background): “In fact, some previous work showed that task similarity helps improve performance in the context of transfer learning [2, 3, 35], while some others claimed that task dissimilarity could help continual learning [13] although their discussion was more related to the permuted MNIST setting.”
Continual and Multi-Task Architecture Search (2019, neutral): “…is primarily used to improve the generalization performance of a task by leveraging knowledge from related tasks (Caru-ana, 1998; Collobert and Weston, 2008; Girshick, 2015; Luong et al., 2015; Ruder et al., 2017; Au-genstein et al., 2018; Guo et al., 2018; Oh et al., 2017; Ruder and Plank, 2017).”
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, background): “Similarly, a series of works has started a systematic empirical analysis of this phenomenon in deep networks (Parisi et al., 2019; Mirzadeh et al., 2020; Neyshabur et al., 2020; Nguyen et al., 2019; Ruder & Plank, 2017).”
Dynamic Data Selection for Neural Machine Translation (2017, methodology): “…work a domain-relevance ranking of the bitext following Axelrod et al. (2011), dynamic data selection can also be applied using other ranking criteria, for example limiting redundancy in the training data (Lewis and Eetemadi, 2013) or complementing similarity with diversity (Ruder and Plank, 2017).”
An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models (2020, neutral): “Similar to our ﬁndings on minority examples, source examples similar to the target data have been found to be helpful to transfer (Ruder and Plank, 2017; Liu et al., 2019a).”
Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimization (2021, methodology): “[49] and Ruder and Plank [37], we use Bayesian Optimization [6] here, which is a framework used to globally optimize any black-box function [41].”
Unsupervised Domain Clusters in Pretrained Language Models (2020, methodology): “Many works used vector-based retrieval for data selection; Ruder and Plank (2017) learn to select data using Bayesian optimization, and explored word2vec for that purpose.”
AdaPrompt: Adaptive Model Training for Prompt-based NLP (2022, background): “, 2018), parsing (Plank and Van Noord, 2011; Ruder and Plank, 2017), named entity recognition (Dai et al.”
Neural Unsupervised Domain Adaptation in NLP—A Survey (2020, methodology): “, 2016) to learn under domain shift (Ruder and Plank, 2017).”
AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning (2019, methodology): “In particular, Ruder and Plank (2017) used Bayesian optimization to select relevant training instances for transfer learning, and Tsvetkov et al. (2016) applied it to learn a curriculum for training word embeddings via reordering data.”
Data Selection for Language Models via Importance Resampling (2023, methodology): “[29; 69; 47] have also been used in selecting data in the domain adaptation setting.”
ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation (2023, methodology): “Similar to (Ruder & Plank, 2017; Allaway et al., 2021), we adopt the Jensen - Shannon ( JS ) divergence between two adjacent domains as an approximation of H -divergence because it has been shown to successfully distinguish domains.”
Ranking Transfer Languages with Pragmatically-Motivated Features for Multilingual Sentiment Analysis (2020, result): “language task (Wang and Neubig, 2019; Do and Gaspers, 2019), motivated by previous studies in domain adaptation (Ruder and Plank, 2017; Plank and van Noord, 2011).”