Multi-task Self-Supervised Visual Learning

By Carl Doersch, Andrew Zisserman. 2017

671 citations recorded by the source corpus.

We investigate methods for combining multiple selfsupervised tasks—i.e., supervised tasks where data can be collected without manual labeling—in order to train a single visual representation. First, we provide an apples-toapples comparison of four different self-supervised tasks using the very deep ResNet-101 architecture. We then combine tasks to jointly train a network. We also explore lasso regularization to encourage the network to factorize the information in its representation, and methods for “harmonizing” network inputs in order to learn a more unified representation. We evaluate all methods on ImageNet classification, PASCAL VOC detection, and NYU depth prediction. Our results show that deeper networks work better, and that combining tasks—even via a na¨ýve multihead architecture—always improves performance. Our best joint network nearly matches the PASCAL performance of a model pre-trained on ImageNet classification, and matches the ImageNet network on NYU depth prediction.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, neutral): “Recent work has jointly trained image classifiers with tasks like relative position and colorization (Doersch and Zisserman, 2017), sequence taggers with language modeling (Rei, 2017), and reinforcement learning agents with predicting changes in the environment (Jaderberg et al.”
A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “Other work explores the use of in-batch samples for negative sampling instead of a memory bank (Doersch & Zisserman, 2017; Ye et al., 2019; Ji et al., 2019).”
Momentum Contrast for Unsupervised Visual Representation Learning (2019, methodology): “AP 75 pre-train RelPos, by [14] Multi-task [14] Jigsaw, by [26] Table 4.”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “Doersch et al. [12] show that motion-based SSL may be combined with other self-supervision cues namely exemplar, colorization, and spatial-context, to pre-train models that perform better than each of these cues individually.”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, methodology): “In linear/MLP evaluation, widely used in for self-supervised representations (Doersch et al., 2015; Zhang et al., 2016; Noroozi & Favaro, 2016; Doersch & Zisserman, 2017), the weights of a pre-trained network are frozen, and a linear layer/MLP is trained on top to predict the labels.”
Learning Representations by Maximizing Mutual Information Across Views (2019, methodology): “, 2014, Doersch and Zisserman, 2017, Noroozi and Favaro, 2016, van den Oord et al., 2018, Hénaff et al., 2019]. The non-CPC results are from updated versions of the models by Kolesnikov et al. [2019]. The (sup) models were fully-supervised, with no self-supervised costs.”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, methodology): “Our method, which learns only from Ima-geNet data using a single unsupervised objective, reaches 70.6% when equipped with a ResNet-101 feature extractor (as for most competing methods [15] but not all [8, 9]).”
Learning Visual Representations with Caption Annotations (2020, background): “At the other extreme of visual representation learning, self-supervised learning methods [7,15,16,18,66,67] do not require annotations and fabricate proxy labels from the data itself.”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “Depth prediction measures how well a network represents geometry, and how well that information can be localized to pixel accuracy [40].”
Representation Learning with Contrastive Predictive Coding (2018, methodology): “We follow the same setup as [36] and use a ResNet v2 101 architecture [37] as the image encoder g enc to extract CPC representations (note that this encoder is not pretrained).”
Unsupervised Feature Learning via Non-parametric Instance Discrimination (2018, methodology): “5%, whereas the accuracy with exemplar CNN [3] is only 31.”
Unsupervised Representation Learning by Predicting Image Rotations (2018, background): “…on other vision tasks, such as object recognition, object detection, and semantic segmentation (Zhang et al., 2016a; Larsson et al., 2016; Zhang et al., 2016b; Larsson et al., 2017; Doersch et al., 2015; Noroozi & Favaro, 2016; Noroozi et al., 2017; Pathak et al., 2016a; Doersch & Zisserman, 2017).”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “In vision, multi-task learning has become popular [22, 40, 56, 100].”
Self-Supervised Learning of Pretext-Invariant Representations (2019, methodology): “Prior pretext tasks include image colorization [8, 28, 35, 36, 77, 78], orientation prediction [18], afﬁne transform prediction [76], predicting contextual image patches [9], re-ordering image patches [5, 19, 46, 48], counting visual primitives [47], or their combinations [10].”
Large Scale Adversarial Representation Learning (2019, background): “Rigorous empirical comparisons of many of these approaches have also been conducted [4, 20].”
Local Aggregation for Unsupervised Learning of Visual Embeddings (2019, background): “Indeed, it has been found that deeper networks better minimizing the loss functions used in such tasks gain little transfer learning performance on object recognition tasks [14].”
Revisiting Self-Supervised Visual Representation Learning (2019, methodology): “Doersch and Zisserman [8] implement four different self-supervision methods and make one single neural network learn all of them in a multi-task setting.”
Learning deep representations by mutual information estimation and maximization (2018, methodology): “…approaches include deep volume-preserving maps (Dinh et al., 2014, 2016), deep clustering (Xie et al., 2016; Chang et al., 2017), noise as targets (NAT, Bojanowski & Joulin, 2017), self-supervised methods (Doersch & Zisserman, 2017), and co-learning (Dosovitskiy et al., 2016; Sajjadi et al., 2016).”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, methodology): “These pretext tasks model different properties of images and have been shown to contain complementary information [16].”