An Analysis of Unsupervised Pre-training in Light of Recent Advances

By T. Paine, Pooya Khorrami, Wei Han, Thomas S. Huang. 2014

56 citations recorded by the source corpus.

Convolutional neural networks perform well on object recognition because of a number of recent advances: rectified linear units (ReLUs), data augmentation, dropout, and large labelled datasets. Unsupervised data has been proposed as another way to improve performance. Unfortunately, unsupervised pre-training is not used by state-of-the-art methods leading to the following question: Is unsupervised pre-training still useful given recent advances? If so, when? We answer this in three parts: we 1) develop an unsupervised method that incorporates ReLUs and recent unsupervised regularization techniques, 2) analyze the benefits of unsupervised pre-training compared to data augmentation and dropout on CIFAR-10 while varying the ratio of unsupervised to supervised samples, 3) verify our findings on STL-10. We discover unsupervised pre-training, as expected, helps when the ratio of unsupervised to supervised samples is high, and surprisingly, hurts when the ratio is low. We also use unsupervised pre-training with additional color augmentation to achieve near state-of-the-art performance on STL-10.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Unsupervised Pretraining for Sequence to Sequence Learning (2016, methodology): “In sequence to sequence learning, an RNN encoder is used to represent x1, ..., xm as a hidden vector, which is given to an RNN decoder to produce the output sequence.”
Self-Supervised Learning of Visual Features through Embedding Images into Text Topic Spaces (2017, neutral): “TextTopicNet (Wiki) - CNN-ﬁnetuning * 76.51% TextTopicNet (Wiki) - fc7+SVM * 66.00% Semi-supervised auto-encoder [44] 74.33 % Convolutional k-means [8] 74.10% CNN with Target Coding [43] 73.15% Exemplar convnets [7] 72.80% Unsupervised pre-training [26] 70.20% Swersky et al .”
A review on weight initialization strategies for neural networks (2021, background): “Paine et al. (2014) have proposed a zero-bias convolutional autoencoder (CAE) to learn the parameters from the input data in an unsupervised way.”
Unsupervised Deep Learning Applied to Breast Density Segmentation and Mammographic Risk Scoring (2016, unknown): “This is in line with several works (e.g., [24], [63]–[65]) that demonstrated the merits of employing unsupervised pre-training with autoencoders in convolutional architectures.”
Training Neural Networks with Local Error Signals (2019, background): “…of the global loss, and this has in certain cases been shown to improve the performance after ﬁne-tuning using global back-propagation (Hinton et al., 2006; Bengio et al., 2007; Salakhutdinov & Hinton, 2009; Erhan et al., 2010; Vincent et al., 2008; Paine et al., 2014; Dong et al., 2018).”
Generative Pretraining From Pixels (2020, background): “Retrospective study of unsupervised pre-training demonstrated that it could even hurt performance in modern settings (Paine et al., 2014).”
Transfer Learning with Deep Convolutional Neural Network for SAR Target Classification with Limited Labeled Data (2017, methodology): “[39] applied ReLUs and some regularization techniques such as the use of zero-bias to CAE, which was proved to achieve superior performance to previous methods.”