Task2Vec: Task Embedding for Meta-Learning

By Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes, Stefano Soatto, Pietro Perona. 2019

377 citations recorded by the source corpus.

We introduce a method to generate vectorial representations of visual classification tasks which can be used to reason about the nature of those tasks and their relations. Given a dataset with ground-truth labels and a loss function, we process images through a "probe network" and compute an embedding based on estimates of the Fisher information matrix associated with the probe network parameters. This provides a fixed-dimensional embedding of the task that is independent of details such as the number of classes and requires no understanding of the class label semantics. We demonstrate that this embedding is capable of predicting task similarities that match our intuition about semantic and taxonomic relations between different visual tasks. We demonstrate the practical value of this framework for the meta-task of selecting a pre-trained feature extractor for a novel task. We present a simple meta-learning framework for learning a metric on embeddings that is capable of predicting which feature extractors will perform well on which task. Selecting a feature extractor with task embedding yields performance close to the best available feature extractor, with substantially less computational effort than exhaustively training and evaluating all available models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Toward Understanding Catastrophic Forgetting in Continual Learning (2019, methodology): “Task2Vec [2] is a recently developed framework for embedding visual classiﬁcation tasks as vectors in a real vector space.”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, methodology): “, 2020), potentially with task2vec methods (Achille et al., 2019; Vu et al., 2020); (3) applying task augmentation strategies to prevent over-fitting (Murty et al.”
Generalized Variational Continual Learning (2020, background): “Like in Achille et al. (2019), this likely could be used as a means of knowing which tasks to learn continually and which tasks to separate (i.e. tasks from the same cluster would likely beneﬁt from joint training, while tasks from different ones should be separately trained), however we did not…”
Meta-Consolidation for Continual Learning (2020, methodology): “A richer embedding such as Task2Vec [1], which takes into account task correlations, may also be used.”
Incremental Meta-Learning via Indirect Discriminant Alignment (2020, methodology): “Hard task mining could be done by selecting tasks using a distance such as [1], by sampling a random class and picking the 4 closest ones in Task2Vec space for a 5-way setup.”
Exploring and Predicting Transferability across NLP Tasks (2020, methodology): “This ranking is evaluated using two metrics: (1) the average rank ρ of the source task with the highest absolute transfer gain from Section 2’s experiments, and (2) the Normalized Discounted Cumulative Gain (NDCG; Järvelin and Kekäläinen, 2002), a common information retrieval measure that evaluat”
Which Tasks Should Be Learned Together in Multi-task Learning? (2019, neutral): “This was followed by a number of recent works, which further analyzed task relationships [30, 13, 1] for transfer learning.”
When Does Self-supervision Improve Few-shot Learning? (2019, background): “Future work could also investigate how and when self-supervision improves generalization across self-supervised and supervised tasks empirically [1, 66].”
Revisiting Meta-Learning as Supervised Learning (2020, methodology): “We expect that applying a more sophisticated and application-dependent metric (e.g., using higher-order statistics (Achille et al., 2019b) or Task2Vec (Achille et al., 2019a)) to characterize task similarity would further improve the performance.”
Transforming task representations to perform novel tasks (2020, methodology): “Achille et al. (36) proposed computing embeddings for visual tasks from the Fisher information of a task-tuned model.”
Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning (2022, background): “Prior work (Achille et al., 2019; Vu et al., 2020) formalizes the notion of task similarity using learned task embeddings so an”
Merging Models with Fisher-Weighted Averaging (2021, methodology): “While other methods (e.g. [1]) exist for estimating the Fisher, we leave their exploration for future work.”
Frustratingly Easy Transferability Estimation (2021, methodology): “To improve the performance on a target task and avoid negative transfer, there have been various works on transferability estimation between tasks [6, 10, 11, 7, 12, 8, 13, 9, 21], which we summarize them in Table 1.”
Transferability and Hardness of Supervised Classification Tasks (2019, methodology): “Some of the answers to these questions established task relationship indices, as in the Taskonomy [69] and Task2Vec [1, 2] projects.”
LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation (2021, background): “…traditional, one-hot encodings requiring the robot to learn from scratch. of defining the number of tasks a priori by leveraging a continuous embedding space [4], [5], [6], [7], these approaches do not incorporate human feedback, knowledge, or specification, thereby ignoring useful prior knowledge.”
Adversarial Fisher Vectors for Unsupervised Representation Learning (2019, methodology): “More recently, Fisher Information has also been applied to tasks such as meta learning [26].”