Transitive Invariance for Self-Supervised Visual Representation Learning

By X. Wang, Kaiming He, Abhinav Kumar Gupta. 2017

183 citations recorded by the source corpus.

Learning visual representations with self-supervised learning has become popular in computer vision. The idea is to design auxiliary tasks where labels are free to obtain. Most of these tasks end up providing data to learn specific kinds of invariance useful for recognition. In this paper, we propose to exploit different self-supervised approaches to learn representations invariant to (i) inter-instance variations (two objects in the same class should have similar features) and (ii) intra-instance variations (viewpoint, pose, deformations, illumination, etc.). Instead of combining two approaches with multi-task learning, we argue to organize and reason the data with multiple variations. Specifically, we propose to generate a graph with millions of objects mined from hundreds of thousands of videos. The objects are connected by two types of edges which correspond to two types of invariance: “different instances but a similar viewpoint and category” and “different viewpoints of the same instance”. By applying simple transitivity on the graph with these edges, we can obtain pairs of images exhibiting richer visual invariance. We use this data to train a Triplet-Siamese network with VGG16 as the base architecture and apply the learned representations to different recognition tasks. For object detection, we achieve 63.2% mAP on PASCAL VOC 2007 using Fast R-CNN (compare to 67.3% with ImageNet pre-training). For the challenging COCO dataset, our method is surprisingly close (23.5%) to the ImageNet-supervised counterpart (24.4%) using the Faster R-CNN framework. We also show that our network can perform significantly better than the ImageNet network in the surface normal estimation task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “As a result, and with the recent emergence of research video data sets [1, 59], videos have been investigated in the context of self-supervision (for example, [40, 64, 63, 28, 65, 73, 41, 51, 42, 3, 2]).”
Unsupervised Feature Learning via Non-parametric Instance Discrimination (2018, methodology): “For videos, self-supervision strategies include: leveraging temporal continuity via tracking [44, 45], predicting future [42], or preserving the equivariance of egomotion [13, 50, 30].”
Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “PIRL is related to approaches that learn invariant image representations via contrastive learning [15, 29, 68, 77, 81], clustering [6, 7, 56, 78], or maximizing mutual information [4, 29, 31].”
A critical analysis of self-supervision, or what we can learn from a single image (2019, methodology): “…(Misra et al., 2016; Wei et al., 2018; Lee et al., 2017; Sermanet et al., 2018), spatio-temporal cues (Isola et al., 2015; Gao et al., 2016; Wang et al., 2017), foreground-background segmentation via video segmentation (Pathak et al., 2017), optical-ﬂow (Gan et al., 2018; Mahendran et al.,…”
Local Aggregation for Unsupervised Learning of Visual Embeddings (2019, methodology): “The typical PASCAL detection task evaluation procedure [7, 76, 72, 69] finetunes unsupervised architectures using the Fast RCNN [21] method.”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, background): “This provides a much stronger baseline (our scratch model outperforms the best numbers reported in [70]).”
Deep Clustering for Unsupervised Learning of Visual Features (2018, background): “We also report the numbers obtained with other unsupervised approaches [13, 64].”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, neutral): “Many self-supervised methods manipulate the input data to extract a supervised signal in the form of a pretext task [1, 13, 26, 28, 30, 36, 38, 41, 42, 47, 48, 57].”
Self-labelling via simultaneous clustering and representation learning (2019, methodology): “, 2018), spatio-temporal cues (Isola et al., 2015; Gao et al., 2016; Wang et al., 2017), foreground-background separation via video segmentation (Pathak et al.”
Interpreting Deep Visual Representations via Network Dissection (2017, methodology): “The self-supervised models all used AlexNet or an AlexNet-derived architecture, with one exception model transinv [49], which uses VGG as the base network.”
Taskonomy: Disentangling Task Transfer Learning (2018, methodology): “Multi-task learning has experienced recent progress and the reported advantages are another support for existence of a useful structure among tasks [93, 100, 50, 76, 73, 50, 18, 97, 61, 11, 66].”
Boosting Self-Supervised Learning via Knowledge Transfer (2018, background): “A natural question is then: How do we design such a task? Some SSL approaches define pretext tasks through explicit desirable invariances of the metric [32, 33, 10, 23] or such that they implicitly require a good object representation [7, 21, 39].”
Improvements to Context Based Self-Supervised Learning (2017, methodology): “Since then, methods have been introduced that use context arrangement of image patches [9, 31], image completion [36], image colorization [48, 47, 24], motion segmentation [35], motion frame ordering [43, 44, 25], object counting [33] and a multi-task ensemble of many models [10, 21].”
Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction. (2018, background): “…such as the boundary of the objects [28, 38], the context of images [25, 36], and the similarity of two patches from an image [6, 15, 33, 34, 48], and the other is the temporal based methods, which exploit temporal connection between frames such as the temporal order of frames [10, 26, 29]…”
Cross Pixel Optical Flow Similarity for Self-Supervised Learning (2018, neutral): “ction audio from video frames [16,17]. [18] train a two stream architecture to classify whether an image and sound clip go together or not. Temporal information is coupled with ego-motion in [19,20]. [21] use videos along with spatial context pretraining [22] to construct an image graph. Transitivit”
Unsupervised Deep Learning by Neighbourhood Discovery (2019, background): “…to exploit some information intrinsically available in the unlabelled training data, such as spatial context (Doersch et al., 2015; Noroozi & Favaro, 2016), spatio-temporal continuity (Wang & Gupta, 2015; Wang et al., 2017), and colour patterns (Zhang et al., 2016; Larsson et al., 2016).”
Leveraging Large-Scale Uncurated Data for Unsupervised Pre-training of Visual Features (2019, background): “Many pretext tasks have been proposed [22, 31, 47, 51], exploiting, amongst others, spatial context [12, 24, 33, 34, 36], cross-channel prediction [27, 28, 55, 56], or the temporal structure of videos [1, 35, 46].”
A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning (2021, methodology): “Unsupervised learning in videos has followed a similar trajectory with earlier methods focusing on predictive tasks based on motion, color and spatiotemporal ordering [29,43,1, 44,78,85,60,84,58,57,21,51,86,66,22,48,91,16,87,70,45], and contrastive objectives with visual [74, 79, 34, 53, 28, 92] and”