LEEP: A New Measure to Evaluate Transferability of Learned Representations

By Cuong V Nguyen, Tal Hassner, Cédric Archambeau, Matthias W. Seeger. 2020

295 citations recorded by the source corpus.

We introduce a new measure to evaluate the transferability of representations learned by classifiers. Our measure, the Log Expected Empirical Prediction (LEEP), is simple and easy to compute: when given a classifier trained on a source data set, it only requires running the target data set through this classifier once. We analyze the properties of LEEP theoretically and demonstrate its effectiveness empirically. Our analysis shows that LEEP can predict the performance and convergence speed of both transfer and meta-transfer learning methods, even for small or imbalanced data. Moreover, LEEP outperforms recently proposed transferability measures such as negative conditional entropy and H scores. Notably, when transferring from ImageNet to CIFAR100, LEEP can achieve up to 30% improvement compared to the best competing method in terms of the correlations with actual transfer accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Memory Efficient Continual Learning with Transformers (2022, background): “ADA-LEEP and ADA-TransRate results with K=4 Adapters show that selective consolidation of Adapters signiﬁcantly improves the performance.”
What to Pre-Train on? Efficient Intermediate Task Selection (2021, background): “Nguyen et al. (2020), Li et al. (2021) and Deshpande et al. (2021) estimate the trans-ferability between classiﬁcation tasks by building an empirical classiﬁer from the source and target task label distribution.”
A Survey on Negative Transfer (2020, background): “To our knowledge, NT was first studied in 2005 [12], and received rapidly increasing attention recently [13], [15], [16].”
OTCE: A Transferability Metric for Cross-Domain Cross-Task Representations (2021, methodology): “We compare the prediction accuracy among OTCE, LEEP and NCE.”
Memory Efficient Continual Learning for Neural Text Classification (2022, methodology): “We observe TransRate performing generally better than LEEP, as in previous experiments.”
Overcoming Negative Transfer: A Survey (2020, methodology): “For negative transfer study, all the three questions have been explored since 2005 [14] and it has raised increasing attention in recent years [17]–[19], such as ﬁnding similar parts of domains, evaluating the trans-ferability of different tasks, models or features, restricting performance gain…”
Zoo-Tuning: Adaptive Transfer from a Zoo of Models (2021, background): “In the face of a zoo of models, recent works seek to predict the transferability of pretrained models to select the best one (Tran et al., 2019; Bao et al., 2019; Nguyen et al., 2020; You et al., 2021).”
LQF: Linear Quadratic Fine-Tuning (2020, unknown): “These results suggest that finding a pre-training closer to the target task may help improve the linearizaion performance [1, 39, 6], which is indeed an intereting direction of research.”
Frustratingly Easy Transferability Estimation (2021, methodology): “DS [11] √ × × √ × [22] √ × × × × [23] √ × × × × NCE [13] √ × √ × × H-Score [10] √ √ × √ × LogME [14] √ √ × √ × LEEP [12] √ √ √ × × TransRate √ √ √ √ √”
A partition-based similarity for classification distributions (2020, methodology): “Our approach, like LEEP (Nguyen et al., 2020), does not require the patterns from the source and target data to be the same but is limited to patterns from the same input space.”
Approximately optimal domain adaptation with Fisher's Linear Discriminant Analysis (2023, background): “Popular task similarity measures utilize information theoretic quantities to evaluate the effectiveness of a pre-trained source model for a particular target task such as H-score (Bao et al., 2018), NCE (Tran et al., 2019), LEEP (Nguyen et al., 2020).”