Transferability and Hardness of Supervised Classification Tasks

By Anh Tuan Tran, Cuong V Nguyen, Tal Hassner. 2019

209 citations recorded by the source corpus.

We propose a novel approach for estimating the difficulty and transferability of supervised classification tasks. Unlike previous work, our approach is solution agnostic and does not require or assume trained models. Instead, we estimate these values using an information theoretic approach: treating training labels as random variables and exploring their statistics. When transferring from a source to a target task, we consider the conditional entropy between two such variables (i.e., label assignments of the two tasks). We show analytically and empirically that this value is related to the loss of the transferred model. We further show how to use this value to estimate task hardness. We test our claims extensively on three large scale data sets---CelebA (40 tasks), Animals with Attributes~2 (85 tasks), and Caltech-UCSD Birds~200 (312 tasks)---together representing 437 classification tasks. We provide results showing that our hardness and transferability estimates are strongly correlated with empirical hardness and transferability. As a case study, we transfer a learned face recognition model to CelebA attribute classification tasks, showing state of the art accuracy for highly transferable attributes.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

Memory Efficient Continual Learning with Transformers (2022, background): “The problem of computing the information carried by a representation in an efficient manner has been already studied in the transfer learning community [3, 58, 56].”
Frustratingly Easy Transferability Estimation (2021, methodology): “DS [11] √ × × √ × [22] √ × × × × [23] √ × × × × NCE [13] √ × √ × × H-Score [10] √ √ × √ × LogME [14] √ √ × √ × LEEP [12] √ √ √ × × TransRate √ √ √ √ √”
Zoo-Tuning: Adaptive Transfer from a Zoo of Models (2021, background): “In the face of a zoo of models, recent works seek to predict the transferability of pretrained models to select the best one (Tran et al., 2019; Bao et al., 2019; Nguyen et al., 2020; You et al., 2021).”
Overcoming Negative Transfer: A Survey (2020, methodology): “Domain Test performance based [22], [31], [32] Selection/ Similarity measure based [18], [33]–[40] Weighting Bayesian Theorem based [13], [41], [42]”
OTCE: A Transferability Metric for Cross-Domain Cross-Task Representations (2021, methodology): “NCE [38] adopts conditional entropy to evaluate transferability and task hardness under a particular setting, i.”
LEEP: A New Measure to Evaluate Transferability of Learned Representations (2020, methodology): “We compare the LEEP measure with the NCE measure proposed by Tran et al. (2019) and the H score proposed by Bao et al. (2019).”
A Survey on Negative Transfer (2020, background): “Fine-Tuning Clustering quality [39] Entropy [16], [40], [41]”
A partition-based similarity for classification distributions (2020, background): “4 RELATED WORK There has been a flurry of recent empirical work (Kifer et al., 2004; Achille et al., 2019; Bao et al., 2018; Tran et al., 2019; Bhattacharjee et al., 2020; Nguyen et al., 2020) defining statistics that help to determine whether or not a particular model or representation learned on a”