AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning

By Ximeng Sun, Rameswar Panda, Rogério Schmidt Feris. 2019

327 citations recorded by the source corpus.

Multi-task learning is an open and challenging problem in computer vision. The typical way of conducting multi-task learning with deep neural networks is either through handcrafting schemes that share all initial layers and branch out at an adhoc point or through using separate task-specific networks with an additional feature sharing/fusion mechanism. Unlike existing methods, we propose an adaptive sharing approach, called AdaShare, that decides what to share across which tasks for achieving the best recognition accuracy, while taking resource efficiency into account. Specifically, our main idea is to learn the sharing pattern through a task-specific policy that selectively chooses which layers to execute for a given task in the multi-task network. We efficiently optimize the task-specific policy jointly with the network weights using standard back-propagation. Experiments on three challenging and diverse benchmark datasets with a variable number of tasks well demonstrate the efficacy of our approach over state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Meta & Continual Learning

Citations in the corpus

DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning (2021, neutral): “For static gating, we also consider a Gumbelsoftmax based gate [34]–unlike DSelect-k this gate cannot control the sparsity level explicitly (see the supplementary for details).”
Model Zoo: A Growing Brain That Learns Continually (2021, methodology): “…has been recognized that this is not suﬃcient; this has given rise to methods for soft-parameter sharing that either design or learn specialized routing architectures (Rosenbaum et al., 2017; Sun et al., 2019; Fernando et al., 2017; Devin et al., 2017; Misra et al., 2016; Vandenhende et al., 2019).”
Which Tasks Should Be Learned Together in Multi-task Learning? (2019, background): “, 2017), and AdaShare (Sun et al., 2019) attempt to learn what to share and what not to.”
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition (2021, methodology): “We use Adam [28] for the policy network and SGD [45] for the recognition network following [56, 44].”
Dynamic Network Quantization for Efficient Video Inference (2021, unknown): “In contrast, we use Gumbel-Softmax Sampling [24] to circumvent this non-differentiability and make our framework fully differentiable, as in [60, 47].”
Uncertainty-Aware Learning for Zero-Shot Semantic Segmentation (2020, background): “As a fundamental problem in computer vision, this task has attracted a lot of attention from the research community and achieved great success along with the development of deep learning in recent years [7, 8, 14, 15, 19, 24, 38, 45, 50, 51, 57, 59, 60, 63, 66].”
Multi-task Learning (2020, methodology): “To further complicate matters, the benefit or detriment induced from co-training relies on many non-trivial decisions including, but not limited to, dataset characteristics, model architecture, hyperparameters, capacity, and convergence (Wu et al., 2020; Vandenhende et al., 2019; Standley et al., 20”
Efficiently Identifying Task Groupings for Multi-Task Learning (2021, background): “A plethora of multi-task methods addressing what parameters to share among tasks in a model have been developed, such as Neural Architecture Search [19, 47, 49, 35, 22, 39], Soft-Parameter Sharing [40, 14, 52], and asymmetric information transfer [31, 44, 32] to improve multi-task performance.”
Contextualizing Multiple Tasks via Learning to Decompose (2021, background): “Learning multiple tasks simultaneously helps extract the shared model inductive bias, such as embedding and partial top-layer classiﬁers [27, 39, 44], which leads to a more compact and generalizable model for discerning over novel instances from those tasks [28].”