Cross-Stitch Networks for Multi-task Learning

By Ishan Misra, Abhinav Shrivastava, Abhinav Kumar Gupta, Martial Hebert. 2016

1,560 citations recorded by the source corpus.

Multi-task learning in Convolutional Networks has displayed remarkable success in the field of recognition. This success can be largely attributed to learning shared representations from multiple supervisory tasks. However, existing multi-task approaches rely on enumerating multiple network architectures specific to the tasks at hand, that do not generalize. In this paper, we propose a principled approach to learn shared representations in ConvNets using multitask learning. Specifically, we propose a new sharing unit: "cross-stitch" unit. These units combine the activations from multiple networks and can be trained end-to-end. A network with cross-stitch units can learn an optimal combination of shared and task-specific representations. Our proposed method generalizes across multiple tasks and shows dramatically improved performance over baseline methods for categories with few training examples.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

An Overview of Multi-task Learning (2018, background): “Unlike these deep models, the cross-stitch network proposed in [23] combines the hidden feature representations of two tasks to construct more powerful hidden feature representations.”
Growing a Brain: Fine-Tuning by Increasing Model Capacity (2017, background): “Transfer learning is related to both multi-task learning [32, 4, 28, 10, 11, 45, 24, 2] and learning novel categories from few examples [49, 19, 21, 35, 51, 22, 5, 13, 50, 47, 31].”
Panoptic Feature Pyramid Networks (2019, background): “In general, using a single network to solve multiple diverse tasks degrades performance [32], but various strategies can mitigate this [29, 42].”
Panoptic Segmentation (2018, background): “With the success of deep learning for many visual recognition tasks, there has been substantial interest in multitask learning approaches that have broad competence and can solve multiple diverse vision problems in a single framework [19, 32, 34].”
Pixelwise Instance Segmentation with a Dynamically Instantiated Network (2017, methodology): “Possible techniques for doing this are suggested by [25] and [39].”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “In vision, multi-task learning has become popular [22, 40, 56, 100].”
Multi-task Self-Supervised Visual Learning (2017, background): “Inspired by recent works that extract information across network layers for the sake of transfer learning [14, 22, 36], we propose a mechanism which allows a network to choose which layers are fed into each task.”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, methodology): “Setup: We use the surface normal estimation task [22], with the evaluation, and dataset splits as formulated in [5, 45, 68].”
12-in-1: Multi-Task Vision and Language Representation Learning (2019, background): “Advances in multi-task learning have been developed in the context of vision [5,20,32,42,52,53], language [10, 25, 26, 31, 37], and robotics [18, 34, 46].”
A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks (2016, background): “For example, Misra et al. (2016) proposed a multi-task learning model to handle different tasks.”
Learning without Forgetting (2016, background): “Network expansion by itself performs better than feature extraction, but neither variant performs as well as LwF on new tasks.”
UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, background): “…depth estimation and semantic segmentation, while [33] train a system for joint detection, pose estimation and region proposal generation; [70] study the effects of sharing information across networks trained for complementary tasks, while more recently [8] propose the introduction of…”
A Survey on Multi-Task Learning (2017, neutral): “, the cross-stitch network proposed in [22] to learn task relations in terms of the hidden feature representation, which is a bit similar to the task relation learning approach introduced later.”
Exploring Shared Structures and Hierarchies for Multiple NLP Tasks (2018, neutral): “k } for task k , ¯ R k could be ﬁnally derived by normalizing these rewards with a parameterized softmax function (Norouzi et al. 2016): where τ is a hyper-parameter that adjusts the variance of ¯ R k .”
Model Zoo: A Growing Brain That Learns Continually (2021, methodology): “…has been recognized that this is not suﬃcient; this has given rise to methods for soft-parameter sharing that either design or learn specialized routing architectures (Rosenbaum et al., 2017; Sun et al., 2019; Fernando et al., 2017; Devin et al., 2017; Misra et al., 2016; Vandenhende et al., 2019).”
Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, background): “Such encoding is typically achieved, for example, via feature fusion [38], branching [24, 40, 34, 57], self-supervision [10], attention [31], or filter grouping [1].”
Learning to Learn without Forgetting By Maximizing Transfer and Minimizing Interference (2018, background): “…the dynamics of weight sharing, it bears conceptual similarity to mixtures of experts (Jacobs et al., 1991) style models for lifelong and multi-task learning (Misra et al., 2016; Riemer et al., 2016b; Aljundi et al., 2017; Fernando et al., 2017; Shazeer et al., 2017; Rosenbaum et al., 2018).”