On Random Weights and Unsupervised Feature Learning

By Andrew M. Saxe, Pang Wei Koh, Zhenghao Chen, Maneesh Bhand, Bipin Suresh, Andrew Y. Ng. 2011

462 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Sanity Checks for Saliency Maps (2018, neutral): “This is a similar architecture to the one considered in [34].”
Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values (2018, background): “Saxe et al. (2011) showed that features learned from CNNs with random weights perform surprisingly well in a downstream classiﬁcation task when fed to a linear classiﬁer.”
SMASH: One-Shot Model Architecture Search through HyperNetworks (2017, methodology): “The method that most resembles our own is that of Saxe et al. [29], who propose to efﬁciently explore various architectures by training only the output layer of convolutional networks with random convolutional weights.”
Exploration by Random Network Distillation (2018, neutral): “Features of randomly initialized neural networks have been extensively studied in the context of supervised learning (Rahimi & Recht, 2008; Saxe et al., 2011; Jarrett et al., 2009; Yang et al., 2015).”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, background): “For example, Saxe et al. (2011) showed that the architecture hyper-parameters of convolutional networks could be selected using random weights in the lower layers of the network (ﬁlters of the convolution).”
Deep roto-translation scattering for object classification (2014, background): “Better results are however obtained with unsupervised training of the weights, using auto-encoders [29].”
Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning (2016, background): “Note that in this randomly initialized case, we still expect above-chance decoding performance, given past theoretical and empirical work with random networks [20, 36, 42].”
N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning (2017, positive): “t our learned architectures outperform those described in Romero et al. (2014) and Hinton et al. (2015). Architecture Search: There has been much work on exploring the design space of neural networks Saxe et al. (2011), Zoph & Le (2016), Baker et al. (2016), Ludermir et al. (2006), Miikkulainen ”
Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity (2016, background): “esult suggests that optimizing the weights of the last layer can serve as a convex proxy for choosing among dierent architectures prior to training. This idea was advocated and tested empirically in [49]. 1 2 Related work Current theoretical understanding of NN learning. Understanding neural network”
Do NLP Models Know Numbers? Probing Numeracy in Embeddings (2019, positive): “Similar results have been shown for images (Saxe et al., 2011): random CNNs are powerful feature extractors.”
Provable Bounds for Learning Some Deep Representations (2013, background): “Finally, note that random neural nets do seem useful in so-called reservoir computing, and more structured neural nets with random edge weights are also interesting in feature learning (Saxe et al., 2011).”
The Importance of Skip Connections in Biomedical Image Segmentation (2016, background): “This follows the findings of [16], where it is shown that even randomly initialized weights can confer a surprisingly large portion of a model’s performance after training only the classifier.”
Avoiding pathologies in very deep networks (2014, background): “The importance of network architectures relative to learning has been examined by Saxe et al. (2011).”
A guide to convolution arithmetic for deep learning (2016, background): “Other kinds of pooling exist, e.g., mean or average pooling, which all share the same idea of aggregating the input locally by applying a non-linearity to the content of some patches (Boureau et al. , 2010a,b, 2011; Saxe et al. , 2011).”
DeepSat: a learning framework for satellite imagery (2015, background): “Each of these pooling architectures has its own advantages and limitations and numerous studies are in place that investigate the effect of different pooling functions on representation power of the model ([31],[30]).”
The loss surface and expressivity of deep convolutional neural networks (2017, background): “…up to the wide layer k are chosen randomly is also in line with recent empirical observations for CNN architectures that one has little loss in performance if the weights of the initial layers are chosen randomly without training (Jarrett et al., 2009; Saxe et al., 2011; Yosinski et al., 2014).”