Cross-Domain Self-Supervised Multi-task Feature Learning Using Synthetic Imagery

By Zhongzheng Ren, Yong Jae Lee. 2017

217 citations recorded by the source corpus.

In human learning, it is common to use multiple sources of information jointly. However, most existing feature learning approaches learn from only a single task. In this paper, we propose a novel multi-task deep network to learn generalizable high-level visual representations. Since multitask learning requires annotations for multiple properties of the same training instance, we look to synthetic images to train our network. To overcome the domain difference between real and synthetic data, we employ an unsupervised feature space domain adaptation method based on adversarial learning. Given an input synthetic RGB image, our network simultaneously predicts its surface normal, depth, and instance contour, while also minimizing the feature space domain differences between real and synthetic data. Through extensive experiments, we demonstrate that our network learns more transferable representations compared to single-task baselines. Our learned representation produces state-of-the-art transfer learning results on PASCAL VOC 2007 classification and 2012 detection.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A critical analysis of self-supervision, or what we can learn from a single image (2019, neutral): “…(Caron et al., 2018), noise-as-targets (Bojanowski & Joulin, 2017), colorization (Zhang et al., 2016; Larsson et al., 2017), generation (Jenni & Favaro, 2018; Ren & Lee, 2018; Donahue et al., 2017), geometry (Dosovitskiy et al., 2016; Gidaris et al., 2018) and counting (Noroozi et al., 2017).”
Self-labelling via simultaneous clustering and representation learning (2019, neutral): “…al., 2019), noise-as-targets (Bojanowski & Joulin, 2017), colorization (Zhang et al., 2016; Larsson et al., 2017), generation (Jenni & Favaro, 2018; Ren & Lee, 2018; Donahue et al., 2017; Donahue & Simonyan, 2019), predictive coding (Oord et al., 2018; Hénaff et al., 2019), geometry (Dosovitskiy…”
Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction. (2018, methodology): “In addition to the quantitative evaluation, previous methods also analyze network kernels and activation maps to provide qualitative visualization results [3, 6, 29, 38].”
Cross Pixel Optical Flow Similarity for Self-Supervised Learning (2018, neutral): “[33] obtain self-supervision from synthetic data and adapt their model to the domain of real images by training against a discriminator.”
When Does Self-supervision Improve Few-shot Learning? (2019, background): “Previous works in the computer vision literature have shown moderate beneﬁts by combining tasks such as edge, normal, and saliency estimation for images, or part segmentation and detection for humans (Kokkinos, 2017; Maninis et al., 2019; Ren & Lee, 2018).”
Self-supervised Spatiotemporal Feature Learning by Video Geometric Transformations (2018, methodology): “Recently, many self-supervised learning methods were proposed to learn image representations [6, 7, 23, 26, 31, 34].”
Self-Supervised Representation Learning by Rotation Feature Decoupling (2019, methodology): “Apart from single task, Doersch and Zisserman [11] and Ren and Lee [44] also consider using several tasks together to obtain performance boost.”
How Useful Is Self-Supervised Pretraining for Visual Tasks? (2020, background): “These methods can take many forms and also rely on a variety of cues such as information across frames in video [49, 41] or across different image modalities [39, 37, 35].”
Evolving Losses for Unsupervised Video Representation Learning (2020, background): “Multi-task self-supervised learning has also shown promising results [10, 32, 48], where tasks are assumed to have equal weights and are not multi-modal.”
MARE: Self-Supervised Multi-Attention REsu-Net for Semantic Segmentation in Remote Sensing (2021, background): “The strategies were several, such as patch context [40,41], in-painting [42], colorization [43,44], jigsaw puzzles [45], noise [46], generation [47], rotation [16]), and counting [48].”
UM-Adapt: Unsupervised Multi-Task Adaptation Using Adversarial Cross-Task Distillation (2019, background): “Another set of work, focuses on learning task-agnostic generalized visual representations utilizing the advances in multi-task learning techniques [61, 45].”
Video Jigsaw: Unsupervised Learning of Spatiotemporal Context for Video Action Recognition (2018, background): “There is also recent work on using synthetic imagery and its ‘free anno-tations’ to learn visual representations [43] by combining multiple self-supervised tasks.”
Learning To Classify Images Without Labels (2020, methodology): “lving jigsaw puzzles [35,37], colorizing images [58,29], using adversarial training [14,15], predicting noise [3], counting [36], predicting rotations [16], spotting artifacts [23], generating images [41], using predictive coding [38,20], performing instance discrimination [51,18,7,48,32], and so on”
Attentive Single-Tasking of Multiple Tasks (2019, background): “…is beneﬁcial for unsupervised learning [45], when tasks provide complementary information (eg. depth boundaries and motion boundaries [70]), in cases where task A acts as regularizer for task B due to limited data [32], or in order to learn more generic representations from synthetic data [49].”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, methodology): “It has been combined with clustering [68], semi-supervised learning [145], multi-task learning [157], [158], transfer learning [159], [160], [161], graph NNs [147], [162], [163], reinforcement learning [164], [165], [166], few-shot learning [167], [168], neural architecture search [169], robust…”
Steering Self-Supervised Feature Learning Beyond Local Pixel Statistics (2020, background): “Some works have explored the combination of different self-supervised tasks via multi-task learning [46, 9].”
Self-Supervised Learning in Remote Sensing: A review (2022, background): “Ren and Lee [154] trained an encoder–decoder network to predict the depth map, surface normal map, and instance contour map from a synthetic image, and a discriminator network is trained to distinguish between features from the synthetic image and the real image.”
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey (2019, methodology): “Only a few work explored self-supervised learning from synthetic data by using GAN to bridge the domain gap [30], [166].”