Deep unsupervised learning through spatial contrasting

By Elad Hoffer, Itay Hubara, Nir Ailon. 2016

23 citations recorded by the source corpus.

Convolutional networks have marked their place over the last few years as the best performing model for various visual tasks. They are, however, most suited for supervised learning from large amounts of labeled data. Previous attempts have been made to use unlabeled data to improve model performance by applying unsupervised techniques. These attempts require different architectures and training methods. In this work we present a novel approach for unsupervised training of Convolutional networks that is based on contrasting between spatial regions within images. This criterion can be employed within conventional neural networks and trained using standard techniques such as SGD and back-propagation, thus complementing supervised methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Scattering Networks for Hybrid Representation Learning (2018, methodology): “3 percent [24] using unsupervised learning and the full training set.”
Matterport3D: Learning from RGB-D Data in Indoor Environments (2017, methodology): “Similar to keypoint matching, we train this model in a triplet Siamese fashion, using the distance ratio loss from [19].”
Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction. (2018, background): “…as supervision, such as the boundary of the objects [28, 38], the context of images [25, 36], and the similarity of two patches from an image [6, 15, 33, 34, 48], and the other is the temporal based methods, which exploit temporal connection between frames such as the temporal order of frames…”
Scaling the Scattering Transform: Deep Hybrid Networks (2017, neutral): “3% reported in [14] using unsupervised learning and the full unlabeled and labeled training set.”
Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching (2017, methodology): “We train the network by feeding it a balanced 1:1 ratio of matching and non-matching image pairs (one observed image and one product image) from the set of known objects, and backpropagate gradients from the distance ratio loss (Triplet loss [32]).”
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey (2019, background): “Some papers refer to this type of learning methods as unsupervised learning [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48].”
Self-supervised Spatiotemporal Feature Learning by Video Geometric Transformations (2018, methodology): “Based on the pretext tasks, these methods fall into two categories: texture-based methods which mainly utilize the texture information of images as the supervision information such as the boundary of the objects [22, 34], the context of images [18, 31], and the similarity of two patches from an imag”
Generative Semantic Manipulation with Contrasting GAN (2017, background): “To tackle such large semantic changes, we propose a novel contrasting GAN (contrast-GAN) in the spirit of learning by comparisons [27, 8].”
Learning Visual Affordances for Robotic Manipulation (2019, methodology): “We train the network by feeding it a balanced 1:1 ratio of matching and non-matching image pairs (one observed image and one product image) from the set of known objects, and backpropagate gradients from the distance ratio loss (Triplet loss [42]).”
Regularization for Deep Learning: A Taxonomy (2017, methodology): “• Initialization without pre-training – Random weight initialization (Rumelhart et al., 1986, p. 330; Glorot and Bengio, 2010; He et al., 2015; Hendrycks and Gimpel, 2016) – Orthogonal weight matrices (Saxe et al., 2013) – Data-dependent weight initialization (Krähenbühl et al., 2015) • Initializati”