Taskonomy: Disentangling Task Transfer Learning

By Amir Zamir, Alexander Sax, Bokui (William) Shen, Bokui (William) Shen, Leonidas J. Guibas, Jitendra Malik, Silvio Savarese. 2018

1,404 citations recorded by the source corpus.

Do visual tasks have a relationship, or are they unrelated? For instance, could having surface normals simplify estimating the depth of an image? Intuition answers these questions positively, implying existence of a structure among visual tasks. Knowing this structure has notable values; it is the concept underlying transfer learning and provides a principled way for identifying redundancies across tasks, e.g., to seamlessly reuse supervision among related tasks or solve many tasks in one system without piling up the complexity. We proposes a fully computational approach for modeling the structure of space of visual tasks. This is done via finding (first and higher-order) transfer learning dependencies across a dictionary of twenty six 2D, 2.5D, 3D, and semantic tasks in a latent space. The product is a computational taxonomic map for task transfer learning. We study the consequences of this structure, e.g. nontrivial emerged relationships, and exploit them to reduce the demand for labeled data. We provide a set of tools for computing and probing this taxonomical structure including a solver users can employ to find supervision policies for their use cases.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, background): “ sentence meaning rather than any task-speciﬁc skill, and because we have seen dramatic successes with pretraining in the related domains of word embeddings (Mikolov et al., 2013) and image encoders (Zamir et al., 2018). More concretely, four recent papers show that pretrained sentence encoders can ”
Do Better ImageNet Models Transfer Better? (2018, background): “Finally, a recent work devised a strategy to transfer when labeled data from many different domains is available [75].”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “d where success greatly depends on making effective use of unlabeled data. Tasks can often be combined to form even more powerful learning signals, allowing larger networks to be trained successfully [15,66,36]. While many of these tasks require predicting ﬁxed, lowlevel aspects of the data, another”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “In vision, multi-task learning has become popular [22, 40, 56, 100].”
Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “Finally, we deploy MCC, trained on Hypersim, on novel scenes from Taskonomy [81].”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “For example, Taskonomy [26] contains 26 tasks ranging from edge detection to vanishing point estimation and scene classification.”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…efficiency (Shin et al., 2020; Gao et al., 2020); (2) learning to select appropriate source tasks to learn from during upstream learning (Za- mir et al., 2018; Standley et al., 2020), potentially with task2vec methods (Achille et al., 2019; Vu et al., 2020); (3) applying task…”
GAN Memory with No Forgetting (2020, methodology): “Distinct from existing methods, our GAN memory leverages transfer learning [6, 16, 46, 92, 58] and (image) style transfer [18, 30, 41].”
Incremental Learning Through Deep Adaptation (2017, background): “For example, the recent Taskonomy work [43] proposes to first analyze the transfer-learning performance between all pairs of a large set of different task.”
Gradient Surgery for Multi-Task Learning (2020, methodology): “…to many different settings, including supervised learning (Zhang et al., 2014; Long & Wang, 2015; Yang & Hospedales, 2016; Sener & Koltun, 2018; Zamir et al., 2018) and reinforcement-learning (Espeholt et al., 2018; Wilson et al., 2007), as well as many different domains, such as vision (Bilen…”
Meta-Transfer Learning for Few-Shot Learning (2018, background): “Transfer learning What and how to transfer are key issues to be addressed in transfer learning, as different methods are applied to different source-target domains and bridge different transfer knowledge [34, 56, 54, 58].”
MultiGrain: a unified image embedding for classes and instances (2019, background): “Multi-task training is an active area of research [27, 47], motivated by the observation that deep neural networks are transferable to a wide range of vision tasks [35].”
Task2Vec: Task Embedding for Meta-Learning (2019, background): “Taskonomy [39] explores the structure of the space of tasks, focusing on the question of effective knowledge transfer in a curated collection of 26 visual tasks, ranging from classiﬁcation to 3D reconstruction, deﬁned on a common domain.”
Exploring and Predicting Transferability across NLP Tasks (2020, methodology): “Recent work on task embeddings in computer vision offers a more principled way to encode tasks for meta-learning (Zamir et al., 2018; Achille et al., 2019; Yan et al., 2020).”
Meta-Learning in Neural Networks: A Survey (2020, background): “object recognition, but ideally we would like to be able to transfer between heterogeneous tasks such as those studied in Taskonomy [289].”
The Effectiveness of Memory Replay in Large Scale Continual Learning (2020, methodology): “Following Zamir et al. (2018), we use 15 -layer ConvNet model with upsampling blocks as the task heads h i .”
Domain Adaptive Transfer Learning with Specialist Models (2018, methodology): “[38] deﬁne a method to compute transferabil-Top 1.”
BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning (2018, methodology): “Real-world applications require performing a combina-∗ tion of perception tasks with different complexities, instead of only homogeneous multiple tasks with the same prediction structure [27, 38, 1, 21].”