Characterizing and Avoiding Negative Transfer

By Zirui Wang, Zihang Dai, Barnab'as P'oczos, Jaime G. Carbonell. 2018

510 citations recorded by the source corpus.

When labeled data is scarce for a specific target task, transfer learning often offers an effective solution by utilizing data from a related source task. However, when transferring knowledge from a less related source, it may inversely hurt the target performance, a phenomenon known as negative transfer. Despite its pervasiveness, negative transfer is usually described in an informal manner, lacking rigorous definition, careful analysis, or systematic treatment. This paper proposes a formal definition of negative transfer and analyzes three important aspects thereof. Stemming from this analysis, a novel technique is proposed to circumvent negative transfer by filtering out unrelated source data. Based on adversarial networks, the technique is highly generic and can be applied to a wide range of transfer learning algorithms. The proposed approach is evaluated on six state-of-the-art deep transfer methods via experiments on four benchmark datasets with varying levels of difficulty. Empirically, the proposed method consistently improves the performance of all baseline methods and largely avoids negative transfer, even when the source data is degenerate.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

How Well Does Self-Supervised Pre-Training Perform with Streaming Data? (2021, background): “These phenomena are due to the so-called “negative transfer” (Wang et al., 2019), which is also discussed in other model pre-training studies (Newell & Deng, 2020; Gururangan et al., 2020).”
Efficient Meta Lifelong-Learning with Limited Memory (2020, background): “…particular, they fail to either effectively reuse previously acquired knowledge to help learn new tasks, or they forget prior skills when learning new ones - these two phenomena are known as negative transfer (Wang et al., 2019) and catastrophic forgetting (McCloskey and Cohen, 1989), respectively.”
A Comprehensive Survey on Transfer Learning (2019, background): “Although there have been some studies on negative transfer, negative transfer still needs further systematic analyses [3].”
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021, background): “where achieving good performance on one task can hinder performance on another (Wang et al., 2019c).”
Transfer learning: a friendly introduction (2022, background): “Although few studies have been done on negative transfers, more systematic research is still needed [62].”
A Balanced and Uncertainty-aware Approach for Partial Domain Adaptation (2020, background): “The joint distribution shift is the actual root to negative transfer [45].”
Universal Source-Free Domain Adaptation (2020, background): “To effectively address the problem of negative-transfer [50], [6] employed a single discriminator to achieve both adversarial adaptation and class-level weighting of source samples.”
Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative (2021, background): “With MT-TARTAN, an additional drawback noted by Wang et al. (2019); Yu et al. (2020) is that multi-tasking can negatively impact task performance compared to isolated training.”
Harmonizing Transferability and Discriminability for Adapting Object Detectors (2020, background): “Nevertheless, pure domain alignment may potentially deteriorate the semantic consistency and result in negative transfer, which has been extensively explored by numerous prior works [58, 64, 34, 27, 5, 57, 10] in image classification and semantic segmentation tasks.”
muNet: Evolving Pretrained Deep Neural Networks into Scalable Auto-tuning Multitask Systems (2022, background): “The success of transfer-learning applications hinge on adequate prior knowledge selection, that avoids common negative-transfer pitfalls [Rosenstein, 2005, Wang et al., 2019].”
Provable Meta-Learning of Linear Representations (2020, background): “We believe this simulation represents a simple instance of the empirically observed phenomena of “negative” transfer (Wang et al., 2019).”
Regularized Hierarchical Policies for Compositional Transfer in Robotics (2019, unknown): “Commonly, it is not straightforward to train a single model jointly across different tasks as the solutions to tasks might not only interfere positively but also negatively [Wang et al., 2018].”
Frustratingly Easy Transferability Estimation (2021, background): “could vary a lot, even with the possibility of negative transfer (Pan & Yang, 2009; Wang et al., 2019; Zhang et al., 2020).”
A Decade Survey of Transfer Learning (2010–2020) (2020, background): “Furthermore, a study [87] ﬁrst derived a novel deﬁnition of negative from three different perspectives, the chosen model, the divergence between the joint distributions, and the size of labeled target data, respectively.”
A Survey on Negative Transfer (2020, background): “, the learning tasks in the two domains are related/similar [14], [15]; 2) domain correlation, i.”
Deep Transfer Learning for Bearing Fault Diagnosis: A Systematic Review Since 2016 (2023, background): “If a priori knowledge is transferred from an unrelated field, the performance of transfer learning can be greatly affected, as not all the knowledge extracted from the source domain effectively improves the performance of the target task [167, 168].”
Overcoming Negative Transfer: A Survey (2020, background): “For negative transfer study, all the three questions have been explored since 2005 [14] and it has raised increasing attention in recent years [17]–[19], such as ﬁnding similar parts of domains, evaluating the trans-ferability of different tasks, models or features, restricting performance gain…”
Larger Norm More Transferable: An Adaptive Feature Norm Approach for Unsupervised Domain Adaptation (2018, methodology): “Inspired by [46], we propose meaningful protocols to evaluate the robustness of a given algorithm especially under the more general partial setting.”
A Review of Domain Adaptation without Target Labels (2019, methodology): “It thereby explicitly avoids negative transfer [176], [177].”