Branched Multi-Task Networks: Deciding what layers to share

By Simon Vandenhende, Bert De Brabandere, Luc Van Gool. 2019

163 citations recorded by the source corpus.

In the context of multi-task learning, neural networks with branched architectures have often been employed to jointly tackle the tasks at hand. Such ramified networks typically start with a number of shared layers, after which different tasks branch out into their own sequence of layers. Understandably, as the number of possible network configurations is combinatorially large, deciding what layers to share and where to branch out becomes cumbersome. Prior works have either relied on ad hoc methods to determine the level of layer sharing, which is suboptimal, or utilized neural architecture search techniques to establish the network design, which is considerably expensive. In this paper, we go beyond these limitations and propose an approach to automatically construct branched multi-task networks, by leveraging the employed tasks' affinities. Given a specific budget, i.e. number of learnable parameters, the proposed approach generates architectures, in which shallow layers are task-agnostic, whereas deeper ones gradually grow more task-specific. Extensive experimental analysis across numerous, diverse multi-tasking datasets shows that, for a given budget, our method consistently yields networks with the highest performance, while for a certain performance threshold it requires the least amount of learnable parameters.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Model Zoo: A Growing Brain That Learns Continually (2021, methodology): “…has been recognized that this is not suﬃcient; this has given rise to methods for soft-parameter sharing that either design or learn specialized routing architectures (Rosenbaum et al., 2017; Sun et al., 2019; Fernando et al., 2017; Devin et al., 2017; Misra et al., 2016; Vandenhende et al., 2019).”
Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, background): “Such encoding is typically achieved, for example, via feature fusion [38], branching [24, 40, 34, 57], self-supervision [10], attention [31], or filter grouping [1].”
Gradient Surgery for Multi-Task Learning (2020, background): “…learning problem based on multiple modules or paths (Fernando et al., 2017; Devin et al., 2016; Misra et al., 2016b; Rusu et al., 2016b; Rosenbaum et al., 2018; Vandenhende et al., 2019; Rosenbaum et al., 2018), or using attention-based architectures (Liu et al., 2018; Maninis et al., 2019).”
Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective (2023, background): “Instead of manually specifying which parameters to share [1, 21, 43] or learning to determine task specific parameters with a combinatorial complexity [41, 51, 56, 59, 61], we assume all parameters in the image encoder of LIQE are shared, whose capacity is dynamically allocated to each task during e”
LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation (2021, background): “…traditional, one-hot encodings requiring the robot to learn from scratch. of defining the number of tasks a priori by leveraging a continuous embedding space [4], [5], [6], [7], these approaches do not incorporate human feedback, knowledge, or specification, thereby ignoring useful prior knowledge.”
Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans (2021, background): “Clearly, such choices do have important effects on the dataset (e.g. see our online demo) and on the resulting models [61, 60, 7, 67]).”
What makes instance discrimination good for transfer learning? (2020, background): “Similar phenomena are observed that task misalignment may lead to negative transfer [33] and that the number of layers that tasks may share depends on the task similarity [31].”
Efficiently Identifying Task Groupings for Multi-Task Learning (2021, background): “A plethora of multi-task methods addressing what parameters to share among tasks in a model have been developed, such as Neural Architecture Search [19, 47, 49, 35, 22, 39], Soft-Parameter Sharing [40, 14, 52], and asymmetric information transfer [31, 44, 32] to improve multi-task performance.”
AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning (2019, methodology): “Only a few methods have attempted to learn multi-branch network architectures, using greedy optimization based on task aﬃnity measures [32, 51] or convolutional ﬁlter grouping [6, 49].”
Sharing Less is More: Lifelong Learning in Deep Networks with Selective Layer Transfer (2020, background): “…work in selective sharing for deep networks has typically either 1.) branched the network into a tree structure (Lu et al., 2017; Yoon et al., 2018; Vandenhende et al., 2019; He et al., 2018), which emphasizes the sharing of lower layers or 2.) introduced new learning modules between task models…”
Multi-Task Learning with Deep Neural Networks: A Survey (2020, methodology): “RSA is a commonly used tool in computational neuroscience to quantitatively compare measures of neural activity, and it has been adopted for analyzing neural network activations by the deep learning community in recent years (Vandenhende et al., 2019).”
Deep Transformers with Latent Depth (2020, background): “In multi-task learning, Multi-Task Attention Network (MTAN) [15], routing network [24] and branched network [28] enables soft parameter sharing by learning a dynamic sub-network for a given task.”
Small Towers Make Big Differences (2020, background): “, 2018), adaptively and dynamically deciding which layers to share for which tasks during the training process (Lu et al., 2017; Vandenhende et al., 2019), or on a higher level, deciding which tasks should be learned together (Standley et al.”
MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning (2020, background): “Despite the progress reported by these or similar works [39, 27, 32, 25, 44], the joint learning of multiple tasks can lead to single-task performance degradation if information sharing happens between unrelated tasks.”