Unsupervised Learning by Predicting Noise

By Piotr Bojanowski, Armand Joulin. 2017

304 citations recorded by the source corpus.

Convolutional neural networks provide visual features that perform remarkably well in many computer vision applications. However, training these networks requires significant amounts of supervision. This paper introduces a generic framework to train deep networks, end-to-end, with no supervision. We propose to fix a set of target representations, called Noise As Targets (NAT), and to constrain the deep features to align to them. This domain agnostic approach avoids the standard unsupervised learning issues of trivial solutions and collapsing of features. Thanks to a stochastic batch reassignment strategy and a separable square loss function, it scales to millions of images. The proposed approach produces representations that perform on par with state-of-the-art unsupervised methods on ImageNet and Pascal VOC.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Emerging Properties in Self-Supervised Vision Transformers (2021, neutral): “Several other works echo this direction, showing that one can train features matching them to a uniform distribution on the `2 hypersphere [5] or by using whitening [19, 66].”
Self-supervised Pretraining of Visual Features in the Wild (2021, methodology): “Self-supervised learning has a long history in computer vision with methods based on autoencoders [42, 51], clustering [1, 5, 11] or instance-level discrimination [3, 15, 21, 52].”
Grafit: Learning fine-grained image representations with coarse labels (2020, methodology): “For this purpose, we build upon recent works [3, 62] that exploit two losses to address both image classification and instance recognition, leveraging the “free” annotations provided by multiple data augmentations of a same instance, in the spirit of self-supervised learning [6, 9, 10, 25].”
Unsupervised Representation Learning by Predicting Image Rotations (2018, neutral): “ImageNet labels from (Bojanowski & Joulin, 2017) 59.7 59.7 Random from (Noroozi & Favaro, 2016) 27.1 12.0 Tracking Wang & Gupta (2015) 38.8 29.8 Context (Doersch et al., 2015) 45.6 30.4 Colorization (Zhang et al., 2016a) 40.7 35.2 Jigsaw Puzzles (Noroozi & Favaro, 2016) 45.3 34.6 BIGAN (Donahue et…”
A critical analysis of self-supervision, or what we can learn from a single image (2019, neutral): “…(Doersch et al., 2015; Noroozi & Favaro, 2016; Noroozi et al., 2018; Mundhenk et al., 2017), clustering (Caron et al., 2018), noise-as-targets (Bojanowski & Joulin, 2017), colorization (Zhang et al., 2016; Larsson et al., 2017), generation (Jenni & Favaro, 2018; Ren & Lee, 2018; Donahue et…”
Learning deep representations by mutual information estimation and maximization (2018, methodology): “It is also similar to noise as targets (Bojanowski & Joulin, 2017), but trains the encoder to match the noise implicitly rather than using a priori noise samples as targets.”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, background): “The label generation can either be domain agnostic [6, 8, 52, 72] or exploit structural properties of the domain, e.”
A Closer Look at Memorization in Deep Networks (2017, background): “Bojanowski & Joulin (2017) show that predicting random noise targets can yield state of the art results in unsupervised learning, corroborating our ﬁndings in Section 3.1, especially Figure 2.”
Unsupervised Pretraining Transfers Well Across Languages (2020, background): “Contrastive approaches are also related to examplar self-supervision [17, 18, 19].”
Deep Clustering for Unsupervised Learning of Visual Features (2018, methodology): “Closer to our work, Bojanowski and Joulin [5] learn visual features on a large dataset with a loss that attempts to preserve the information flowing through the network [37].”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, neutral): “Instance-level classification considers each image in a dataset as its own class [4, 15, 49].”
Self-labelling via simultaneous clustering and representation learning (2019, methodology): “…domain, we ﬁnd a spectrum of methods that either give each data point a unique label (Wu et al., 2018; Dosovitskiy et al., 2016) or train on a ﬂexible number of labels with k-means (Caron et al., 2018), with mutual information (Ji et al., 2018) or with noise (Bojanowski & Joulin, 2017).”
Barlow Twins: Self-Supervised Learning via Redundancy Reduction (2021, methodology): “Noise As Targets This method (Bojanowski & Joulin, 2017) learns to map samples to ﬁxed random targets on the unit sphere, which can be interpreted as a form of whitening.”
Cross-Task Weakly Supervised Learning From Instructional Videos (2019, background): “, minimizing the L2 loss with linear model [3, 2]) or an unrestricted classifier, such as a deep network, but no constraints [4, 7].”
AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations Rather Than Data (2019, methodology): “ImageNet Labels [3] For the following ImageNet experiments, many existing methods have been compared in literature with the same unsupervised AlexNet architecture as well as the classiﬁers upon it for the evaluation.”
Improvements to Context Based Self-Supervised Learning (2017, neutral): “9 Bojanowski [2] 65.”
Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction. (2018, background): “Different types of image transformations are designed as supervision information to train 2DCNNs for image representation learning including image colorization [25], image rotation [12], and image denoise [2].”
Cross Pixel Optical Flow Similarity for Self-Supervised Learning (2018, neutral): “et al. [2] Video-seg [2],Self 61.0 50.2 - Wang et al. [13] Track + Rank [41,13] 63.1 47.5 - Ours Opticalow Self 64.4 50.3 41.4 es Ours direct reg. Opticalow Self 61.4 44.0 37.1 cues Bojanowski et al. [34] - [34] 65.3 49.4 - Doersch et al. [57] Context [31] 65.3 51.1 - Donahue et al. [31] - [31] 60.3”