Representation Similarity Analysis for Efficient Task Taxonomy & Transfer Learning

By Kshitij Dwivedi, Gemma Roig. 2019

172 citations recorded by the source corpus.

Transfer learning is widely used in deep neural network models when there are few labeled examples available. The common approach is to take a pre-trained network in a similar task and finetune the model parameters. This is usually done blindly without a pre-selection from a set of pre-trained models, or by finetuning a set of models trained on different tasks and selecting the best performing one by cross-validation. We address this problem by proposing an approach to assess the relationship between visual tasks and their task-specific models. Our method uses Representation Similarity Analysis (RSA), which is commonly used to find a correlation between neuronal responses from brain data and models. With RSA we obtain a similarity score among tasks by computing correlations between models trained on different tasks. Our method is efficient as it requires only pre-trained models, and a few images with no further training. We demonstrate the effectiveness and efficiency of our method to generating task taxonomy on Taskonomy dataset. We next evaluate the relationship of RSA with the transfer learning performance on Taskonomy tasks and a new task: Pascal VOC semantic segmentation. Our results reveal that models trained on tasks with higher similarity score show higher transfer learning performance. Surprisingly, the best transfer learning result for Pascal VOC semantic segmentation is not obtained from the pre-trained model on semantic segmentation, probably due to the domain differences, and our method successfully selects the high performing models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, methodology): “We then use the Representation Similarity Analysis (RSA), proposed in [11] for transfer learning, as a means to quantify the correlation of the gradients amongst the different tasks.”
Which Tasks Should Be Learned Together in Multi-task Learning? (2019, neutral): “This was followed by a number of recent works, which further analyzed task relationships [30, 13, 1] for transfer learning.”
Collaboration by Competition: Self-coordinated Knowledge Amalgamation for Multi-talent Student Learning (2020, methodology): “(1) 4 https://github.com/kshitijd20/RSA-CVPR19-release The MIA takes as input the stacked representation of the teacher network and generate the target representation σ(l)(t) via passing message from the feature maps of other tasks as follows, MIA ( A (l) T1 (t), · · · , A(l)Tm(t) ) = C m∑ i W (l) i”
Multi-Task Learning for Dense Prediction Tasks: A Survey (2020, methodology): “Similarly, Dwivedi and Roig [62]”
Multi-Task Learning with Deep Neural Networks: A Survey (2020, methodology): “The underlying assumption behind the RSA transfer model in (Dwivedi and Roig, 2019) is that if two tasks would exhibit positive transfer, then single-task networks trained on each of them will tend to learn similar representations, and so RSA will be an accurate measure of the transfer affinities…”
Multi-task Learning (2020, background): “…approach may leverage recent advances in transfer learning focused on understanding task relationships (Zamir et al., 2018; Achille et al., 2019b; Dwivedi & Roig, 2019; Zhuang et al., 2020; Achille et al., 2019a); however, Standley et al. (2019) show transfer learning algorithms which determine…”
Efficiently Identifying Task Groupings for Multi-Task Learning (2021, background): “2 , 15] which also satisﬁes the second condition of Proposition 1, namely cos( a, c ) ≤ η For this choice of g c and as a result of Proposition 1, the fact that g b reduces the loss L a more so than g c ( 6 .”
Branched Multi-Task Networks: Deciding what layers to share (2019, methodology): “the number of tasks, and we opted for a more efficient alternative proposed by [10].”
Frustratingly Easy Transferability Estimation (2021, background): “To improve the performance on a target task and avoid negative transfer, there have been various works on transferability estimation between tasks [6, 10, 11, 7, 12, 8, 13, 9, 21], which we summarize them in Table 1.”