An Information-Theoretic Approach to Transferability in Task Transfer Learning

By Yajie Bao, Yongni Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, Leonidas J. Guibas. 2019

162 citations recorded by the source corpus.

Task transfer learning is a popular technique in image processing applications that uses pre-trained models to reduce the supervision cost of related tasks. An important question is to determine task transferability, i.e. given a common input domain, estimating to what extent representations learned from a source task can help in learning a target task. Typically, transferability is either measured experimentally or inferred through task relatedness, which is often defined without a clear operational meaning. In this paper, we present a novel metric, H-score, an easily-computable evaluation function that estimates the performance of transferred representations from one task to another in classification problems using statistical and information theoretic principles. Experiments on real image data show that our metric is not only consistent with the empirical transferability measurement, but also useful to practitioners in applications such as source model selection and task transfer curriculum learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Memory Efficient Continual Learning with Transformers (2022, background): “The problem of computing the information carried by a representation in an efficient manner has been already studied in the transfer learning community [3, 58, 56].”
Frustratingly Easy Transferability Estimation (2021, methodology): “DS [11] √ × × √ × [22] √ × × × × [23] √ × × × × NCE [13] √ × √ × × H-Score [10] √ √ × √ × LogME [14] √ √ × √ × LEEP [12] √ √ √ × × TransRate √ √ √ √ √”
Zoo-Tuning: Adaptive Transfer from a Zoo of Models (2021, background): “In the face of a zoo of models, recent works seek to predict the transferability of pretrained models to select the best one (Tran et al., 2019; Bao et al., 2019; Nguyen et al., 2020; You et al., 2021).”
LEEP: A New Measure to Evaluate Transferability of Learned Representations (2020, methodology): “We compare the LEEP measure with the NCE measure proposed by Tran et al. (2019) and the H score proposed by Bao et al. (2019).”
OTCE: A Transferability Metric for Cross-Domain Cross-Task Representations (2021, methodology): “We emphasize that the cross-domain cross-task setting is more challenging compared to previous settings that require shared input data or same domain, since both domain difference and task difference deteriorate the transfer performance [27, 5, 26].”
Overcoming Negative Transfer: A Survey (2020, background): “Bao et al. [78] presented a H-score metric, a computable evaluation function that estimates the performance of transferred representations from one task to another in classiﬁcation problems using statistical and information theoretic principles. where f ( X ) be a zero-mean feature function.”