Augmenting Supervised Neural Networks with Unsupervised Objectives for Large-scale Image Classification

By Yuting Zhang, Kibok Lee, Honglak Lee. 2016

145 citations recorded by the source corpus.

Unsupervised learning and supervised learning are key research topics in deep learning. However, as high-capacity supervised neural networks trained with a large amount of labels have achieved remarkable success in many computer vision tasks, the availability of large-scale labeled images reduced the significance of unsupervised learning. Inspired by the recent trend toward revisiting the importance of unsupervised learning, we investigate joint supervised and unsupervised learning in a large-scale setting by augmenting existing neural networks with decoding pathways for reconstruction. First, we demonstrate that the intermediate activations of pretrained large-scale classification networks preserve almost all the information of input images except a portion of local spatial details. Then, by end-to-end training of the entire augmented architecture with the reconstructive objective, we show improvement of the network performance for supervised tasks. We evaluate several variants of autoencoders, including the recently proposed "what-where" autoencoder that uses the encoder pooling switches, to study the importance of the architecture design. Taking the 16-layer VGGNet trained under the ImageNet ILSVRC 2012 protocol as a strong baseline for image classification, our methods improve the validation-set accuracy by a noticeable margin.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Optimization & Training

Citations in the corpus

Learning from Simulated and Unsupervised Images through Adversarial Training (2016, background): “Large labeled training datasets are becoming increasingly important with the recent rise in high capacity deep neural networks [4, 20, 48, 48, 1, 24, 17].”
Densely Connected Convolutional Networks (2016, neutral): “Further, we significantly outperform the current state-ofthe-art results on most of the benchmark tasks.”
A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks (2016, background): “Auxiliary decoders are sometimes known to increase classiﬁcation performance (Zhang et al., 2016).”
Stacked Generative Adversarial Networks (2016, methodology): “Also, some auto-encoder architectures employ layer-wise reconstruction loss [60, 48, 67, 66].”
Early Methods for Detecting Adversarial Images (2016, background): “Auxiliary decoders are sometimes known to increase classiﬁcation performance (Zhang et al., 2016).”
Recent advances in convolutional neural networks (2015, background): “Speciﬁcally, stochastic pooling ﬁrst computes the probabilities p for each region R j by normalizing the activations within the region, i.e ., p i = a i / k j ( a k ) .”
Learning to Navigate in Complex Environments (2016, background): “Recently, the incorporation of additional objectives, designed to augment representation learning through auxiliary reconstructive decoding pathways (Zhang et al., 2016; Rasmus et al., 2015; Zhao et al., 2015; Mirowski et al., 2010), has yielded beneﬁts in large scale classiﬁcation tasks.”
Classification-Reconstruction Learning for Open-Set Recognition (2018, methodology): “Additionally, in higherlevel layers of supervised deep nets, details of input tend to be lost [50, 6] , which may not be preferable in unknown detection.”
High-quality hyperspectral reconstruction using a spectral prior (2017, methodology): “It has been successfully applied for object retrieval [Leng et al. 2015], image classification [Zhang et al. 2016], denoising [Du et al. 2016], or real-time correction of multipath interference in time-of-flight imaging [Marco et al. 2017].”
Invariant Representations through Adversarial Forgetting (2019, methodology): “Inspired by (1) the discovery of richer features in DNN classiﬁers upon augmentation with reconstruction objectives (Sabour, Frosst, and Hinton 2017; Zhang, Lee, and Lee 2016), and (2) the forgetting operation in Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber 1997) cells, the proposed…”
CKD: Cross-Task Knowledge Distillation for Text-to-Image Synthesis (2020, background): “These models have strong capabilities to produce hierarchical representations, which can be verified by the researches of inverting the representations [15]–[17].”
Deep Supervised Learning Using Local Errors (2017, background): “In some of these network configurations, the unsupervised error terms are local to each layer (Zhang et al., 2016).”
Generalizing Skills with Semi-Supervised Reinforcement Learning (2016, methodology): “Semi-supervised learning has been performed with deep models, either by blending unsupervised and supervised objectives (Rasmus et al., 2016; Zhang et al., 2016) or by using generative models, with the labels treated as missing data (Kingma et al., 2014).”
Adapting Auxiliary Losses Using Gradient Similarity (2018, methodology): “Auxiliary tasks have been shown to work well in practice (e.g., Zhang et al., 2016; Jaderberg et al., 2017; Mirowski et al., 2017; Papoudakis et al., 2018).”