Which Tasks Should Be Learned Together in Multi-task Learning?

By Trevor Scott Standley, Amir Zamir, Dawn Chen, Leonidas J. Guibas, Jitendra Malik, Silvio Savarese. 2019

689 citations recorded by the source corpus.

Many computer vision applications require solving multiple tasks in real-time. A neural network can be trained to solve multiple tasks simultaneously using multi-task learning. This can save computation at inference time as only a single network needs to be evaluated. Unfortunately, this often leads to inferior overall performance as task objectives can compete, which consequently poses the question: which tasks should and should not be learned together in one network when employing multi-task learning? We study task cooperation and competition in several different learning settings and propose a framework for assigning tasks to a few neural networks such that cooperating tasks are computed by the same neural network, while competing tasks are computed by different networks. Our framework offers a time-accuracy trade-off and can produce better accuracy using less inference time than not only a single large multi-task neural network but also many single-task networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Meta & Continual Learning

Citations in the corpus

12-in-1: Multi-Task Vision and Language Representation Learning (2019, background): “[41] studies how different vision tasks are related to each other.”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “…are closely related to the multitask learning literature, where the relationship between task similarity and network performance in terms of accuracy and time has been studied (Zhang & Yang, 2021; Ruder, 2017; Standley et al., 2020; Liu, Johns, & Davison, 2019; Kendall, Gal, & Cipolla, 2018).”
One Size Fits All for Semantic Shifts: Adaptive Prompt Tuning for Continual Learning (2023, background): “Addressing this issue, semantic-based task grouping methods (Zamir et al., 2018; Standley et al., 2020; Fifty et al., 2021; Song et al., 2022; Liu et al., 2023) have been introduced, which emphasize clustering semantically similar tasks to amplify positive transfer and boost performance.”
Contextualizing Multiple Tasks via Learning to Decompose (2021, background): “In standard multi-task learning, the correspondence between an example with one of the multiple tasks is explicit [24, 42, 8].”
MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning (2020, background): “detection and classification [9,34], detection and segmentation [6,12], segmentation and depth estimation [7,47], while for the latter recent efforts point to that direction [42].”
Collaboration by Competition: Self-coordinated Knowledge Amalgamation for Multi-talent Student Learning (2020, result): “This was followed by a number of recent works, which further analyzed task relationships [1, 5, 24, 17, 23, 22] for transfer learning.”
Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task (2021, background): “Standley et al. (2020) conduct an empirical study on computer vision tasks for MTL.”
Small Towers Make Big Differences (2020, background): “, 2019), or on a higher level, deciding which tasks should be learned together (Standley et al., 2019).”
Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning (2022, methodology): “Comparing Lw-gen with Lgen + Ldisc, the generator training objective is not directly impacted by the discriminative objective, thus avoiding the gradient interference issue in multi-task learning (Standley et al., 2019)—the gradient for optimizing the generative probability p(x|yl) will be interfere”
Multi-Task Reinforcement Learning with Context-based Representations (2021, background): “In general, choosing which information/knowledge to transfer across the tasks, or even deciding which tasks should be learned together is an open problem (Standley et al., 2019).”
PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning (2020, methodology): “Distinct with multi-task PLATO, the separate design of fine-grained generation and evaluation enables the model to concentrate more on its corresponding task, getting exempt from multi-task disturbance (Standley et al., 2020).”
What makes instance discrimination good for transfer learning? (2020, background): “In [28], task cooperation and task competition are quantitatively measured to improve transfer learning.”
LEEP: A New Measure to Evaluate Transferability of Learned Representations (2020, methodology): “Aside from transfer and meta-transfer learning, our LEEP scores are potentially useful for continual learning (Zenke et al., 2017; Nguyen et al., 2019), multi-task learning (Misra et al., 2016; Standley et al., 2019), and feature selection (Yosinski et al., 2014).”
Transformer is All You Need: Multimodal Multitask Learning with a Unified Transformer (2021, neutral): “[17, 67, 54, 53, 66]), language ( e.g .”
Efficiently Identifying Task Groupings for Multi-Task Learning (2021, methodology): “Following the experimental setup of [46], we evaluate the capacity of TAG to select task groupings on the “Semantic Segmentation”, “Depth Estimation”, “Keypoint Detection”, “Edge Detection”, and “Surface Normal Prediction” objectives in Taskonomy.”
PIP: Physical Interaction Prediction via Mental Imagery with Span Selection (2021, background): “This is a common problem in multi-task learning [56].”
Multi-task learning for natural language processing in the 2020s: where are we going? (2020, background): “A host of other researchers have gone on to argue that task re-latedness plays a key role in determining how knowledge information is shared (Mou et al., 2016; Ben-David and Borbely, 2008) and when an auxiliary task will help and when it will hurt (Standley et al., 2019; Kim et al., 2019).”
Pareto Multi-Task Learning (2019, background): “Current work shows that the adaptive gradient methods would have inferior performance on some tasks [49].”