Frustratingly Easy Transferability Estimation

By Long-Kai Huang, Ying Wei, Yu Rong, Qiang Yang, Junzhou Huang. 2021

75 citations recorded by the source corpus.

Transferability estimation has been an essential tool in selecting a pre-trained model and the layers in it for transfer learning, to transfer, so as to maximize the performance on a target task and prevent negative transfer. Existing estimation algorithms either require intensive training on target tasks or have difficulties in evaluating the transferability between layers. To this end, we propose a simple, efficient, and effective transferability measure named TransRate. Through a single pass over examples of a target task, TransRate measures the transferability as the mutual information between features of target examples extracted by a pre-trained model and their labels. We overcome the challenge of efficient mutual information estimation by resorting to coding rate that serves as an effective alternative to entropy. From the perspective of feature representation, the resulting TransRate evaluates both completeness (whether features contain sufficient information of a target task) and compactness (whether features of each class are compact enough for good generalization) of pre-trained features. Theoretically, we have analyzed the close connection of TransRate to the performance after transfer learning. Despite its extraordinary simplicity in 10 lines of codes, TransRate performs remarkably well in extensive evaluations on 32 pre-trained models and 16 downstream tasks.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Memory Efficient Continual Learning with Transformers (2022, methodology): “ADA-LEEP and ADA-TransRate results with K=4 Adapters show that selective consolidation of Adapters signiﬁcantly improves the performance.”
Continual Learning with Transformers for Image Classification (2022, methodology): “To see the details how the knowledge transfer from a source task to a target task is measured, please see [13].”
A Survey on Negative Transfer (2020, background): “Fine-Tuning Clustering quality [39] Entropy [16], [40], [41]”
Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models (2023, methodology): “On text transcriptions and PubMed 20k datasets containing more than 8 hours of audio data of common medical symptom descriptions,experiments were conducted to compare co-sine similarity and TransRate metrics with multiple data enhancement methods [30].”
Memory Efficient Continual Learning for Neural Text Classification (2022, methodology): “We observe TransRate performing generally better than LEEP, as in previous experiments.”
Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models (2023, methodology): “On text transcriptions and PubMed 20k datasets containing more than 8 hours of audio data of common medical symptom descriptions,experiments were conducted to compare cosine similarity and TransRate metrics with multiple data enhancement methods [33].”