Data-dependent Initializations of Convolutional Neural Networks

By Philipp Krahenbuhl, Carl Doersch, Jeff Donahue, Trevor Darrell. 2015

207 citations recorded by the source corpus.

Convolutional Neural Networks spread through computer vision like a wildfire, impacting almost all visual tasks imaginable. Despite this, few researchers dare to train their models from scratch. Most work builds on one of a handful of ImageNet pre-trained models, and fine-tunes or adapts these for specific tasks. This is in large part due to the difficulty of properly initializing these networks from scratch. A small miscalibration of the initial weights leads to vanishing or exploding gradients, as well as poor convergence properties. In this work we present a fast and simple data-dependent initialization procedure, that sets the weights of a network such that all units in the network train at roughly the same rate, avoiding vanishing or exploding gradients. Our initialization matches the current state-of-the-art unsupervised or self-supervised pre-training methods on standard computer vision tasks, such as image classification and object detection, while being roughly three orders of magnitude faster. When combined with pre-training methods, our initialization significantly outperforms prior work, narrowing the gap between supervised and unsupervised pre-training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017, background): “Most of these works have considered good random initial-izations, though a number of papers have addressed data-dependent initializers (Kr¨ahenb¨uhl et al., 2016; Salimans & Kingma, 2016), including learned initializations (Husken & Goerick, 2000; Maclaurin et al., 2015).”
What makes ImageNet good for transfer learning? (2016, methodology): “t (SUN-CLS). For PASCAL-DET, we used the PASCAL VOC 2007 train/val for ﬁnetuning using the experimental setup and Pre-trained Dataset PASCAL SUN Original 58.3 52.2 127 Classes 55.5 48.7 Random 41.3 [21] 35.7 [2] Table 1: The transfer performance of a network pre-trained using 127 (coarse) classes o”
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks (2016, methodology): “Independently from our work, this type of initialization was recently proposed by different authors [20, 14] who found such data-based initialization to work well for use with the standard parameterization in terms of w .”
Contrastive Multiview Coding (2019, methodology): “To our best knowledge, our CMC, together with concurrent CPC++ [22] and Deep InfoMax++ [3], are the ﬁrst batch of unsupervised/self-supervised learning methods that surpass supervised AlexNet on ImageNet classiﬁcation task.”
Self-Supervised Learning of Visual Features through Embedding Images into Text Topic Spaces (2017, background): “pares our results for image classiﬁcation and object detection on PASCAL with different self-supervised learning algorithms. Method classif. detection TextTopicNet 55.7 43.0 Sound [25] - 44.1 K-means [20] 56.6 45.6 Tracking [41] 62.8 47.4 Patch pos. [6] 55.3 46.6 Egomotion [1] 52.9 41.8 ImageNet [21”
Learning Features by Watching Objects Move (2016, neutral): “19.1 19.7 20.3 20.9 19.6 NA Sup. Masks (Ours) 51.7 51.8 52.7 52.2 52.0 47.5 13.6 13.8 15.5 17.6 18.1 15.1 NA Unsupervised Jigsawz [30] 49.0 50.0 48.9 47.7 45.8 37.1 5.9 8.7 8.8 10.1 9.9 7.9 NA Kmeans [23] 42.8 42.2 40.3 37.1 32.4 26.0 4.1 4.9 5.0 4.5 4.2 4.0 0 Egomotion [2] 37.4 36.9 34.4 28.9 24.1 ”
Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles (2016, methodology): “[21] showed that even with a new data initialization their performance was lower than that of [9,34].”
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (2018, methodology): “Also, the performance of the stacked k-means method [24], shown in yellow, outperforms low-level metrics.”
Local Aggregation for Unsupervised Learning of Visual Embeddings (2019, background): “Over many decades of work, substantial progress has been achieved using a wide variety of unsupervised learning approaches [6, 67, 71, 36, 13, 15, 70, 48, 63, 49].”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, methodology): “We view self-supervised learning as a way to learn feature representations rather than an ‘initialization method’ [34] and thus perform limited fine-tuning of the features.”
Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction (2016, methodology): “These results suggest that split-brain autoencoding is effective not just on Lab images, but also on RGB-D data.”
Adversarial Feature Learning (2016, methodology): “We report results on each of these tasks in Table 3, comparing BiGANs with contemporary approaches to unsupervised [4, 17] and weakly supervised [1, 30] feature learning in the visual domain, as well as the baselines discussed in Section 4.”
Representation Learning by Learning to Count (2017, methodology): “racting the mean intensity value and dividing the zero-mean images by their standard deviation. Method Ref Class. Det. Segm. Supervised [20] [43] 79.9 56.8 48.0 Random [33] 53.3 43.4 19.8 Context [9] [19] 55.3 46.6 - Context [9] [19] 65.3 51.1 - Jigsaw [30] [30] 67.6 53.2 37.6 ego-motion [1] [1] 52.”