Efficiently Identifying Task Groupings for Multi-Task Learning

By Christopher Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, Chelsea Finn. 2021

332 citations recorded by the source corpus.

Multi-task learning can leverage information learned by one task to benefit the training of other tasks. Despite this capacity, naively training all tasks together in one model often degrades performance, and exhaustively searching through combinations of task groupings can be prohibitively expensive. As a result, efficiently identifying the tasks that would benefit from training together remains a challenging design question without a clear solution. In this paper, we suggest an approach to select which tasks should train together in multi-task learning models. Our method determines task groupings in a single run by training all tasks together and quantifying the effect to which one task's gradient would affect another task's loss. On the large-scale Taskonomy computer vision dataset, we find this method can decrease test loss by 10.0% compared to simply training all tasks together while operating 11.6 times faster than a state-of-the-art task grouping method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, methodology): “Aside from the NLP domain, Fifty et al. (2021) proposed a method to select which tasks to transfer to based on task gradients.”
TIES-Merging: Resolving Interference When Merging Models (2023, background): “Moreover, it can be complex and resource-intensive to determine how best to mix datasets to ensure that multitask training is beneficial for all tasks [55, 54, 80, 52, 2, 17].”
BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers (2022, methodology): “However, the jointly trained model does not perform as well as individually trained models for road and lane segmentation, which is a common phenomenon called negative transfer [11, 13] in multi-task learning.”
One Size Fits All for Semantic Shifts: Adaptive Prompt Tuning for Continual Learning (2023, background): “Addressing this issue, semantic-based task grouping methods (Zamir et al., 2018; Standley et al., 2020; Fifty et al., 2021; Song et al., 2022; Liu et al., 2023) have been introduced, which emphasize clustering semantically similar tasks to amplify positive transfer and boost performance.”
PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images (2022, background): “Recently, some other studies [7, 39] also pointed out that not all the 3D computer vision tasks can benefit from other tasks in the joint training setting.”