ScaleNet: An Unsupervised Representation Learning Method for Limited Information

By Huili Huang, M. Mahdi Roozbahani. 2023

760 citations recorded by the source corpus.

Although large-scale labeled data are essential for deep convolutional neural networks (ConvNets) to learn high-level semantic visual representations, it is time-consuming and impractical to collect and annotate large-scale datasets. A simple and efficient unsupervised representation learning method named ScaleNet based on multi-scale images is proposed in this study to enhance the performance of ConvNets when limited information is available. The input images are first resized to a smaller size and fed to the ConvNet to recognize the rotation degree. Next, the ConvNet learns the rotation-prediction task for the original size images based on the parameters transferred from the previous model. The CIFAR-10 and ImageNet datasets are examined on different architectures such as AlexNet and ResNet50 in this study. The current study demonstrates that specific image features, such as Harris corner information, play a critical role in the efficiency of the rotation-prediction task. The ScaleNet supersedes the RotNet by ~7% in the limited CIFAR-10 dataset. The transferred parameters from a ScaleNet model with limited data improve the ImageNet Classification task by about 6% compared to the RotNet model. This study shows the capability of the ScaleNet method to improve other cutting-edge models such as SimCLR by learning effective features for classification tasks.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

S4L: Self-Supervised Semi-Supervised Learning (2019, background): “Finally, [3] proposes a learning procedure that alternates between clustering images in the representation space and learning a model that assigns images to their clusters.”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “DeepCluster [17] partially answers this question.”
VirTex: Learning Visual Representations from Textual Annotations (2020, background): “Early works on self-supervised learning proposed hand-crafted pretext tasks, such as context prediction [15], colorization [17, 18], solving jigsaw puzzles [47], predicting rotation [19], inpainting [16], clustering [27], and generative modeling [48].”
Self-Supervised Learning of Pretext-Invariant Representations (2019, methodology): “We also plan to investigate combinations of PIRL with clustering-based approaches [6, 7].”
Revisiting Self-Supervised Visual Representation Learning (2019, methodology): “Another way to create class labels is to use clustering of the images [2].”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, methodology): “[75] so that we can draw comparisons to existing works (and re-evaluate the model from [8]).”
CrossTransformers: spatially-aware few-shot transfer (2020, background): “Our work on SimCLR episodes inherits from a line of self-supervised learning research, which typically deal with transfer from pretext tasks to semantic ones and must therefore represent more than their training data [3, 13, 15, 21, 24, 30, 36, 47, 60, 101, 102].”
S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning (2023, background): “Different pretext tasks have been proposed like image rotations [17], image colourization [20], clustering [6], and solving jigsaw puzzles from image patch permutations [28].”
Self-Supervised Training Enhances Online Continual Learning (2021, background): “Different pretext tasks have been proposed for visual representation learning including the prediction of image colorization [72], image rotation [24], and several others [50, 20, 19, 8].”
Video Representation Learning by Dense Predictive Coding (2019, neutral): “In contrast to the context-based idea, another set of pretext tasks include carefully designed image-level classification, such as rotation [7] or pseudo-labels from clustering [3].”
MultiGrain: a unified image embedding for classes and instances (2019, methodology): “As an additional contribution, we show that MultiGrain can be learned using only class-level labels via self-supervised learning (Caron et al., 2018).”
Unsupervised Learning From Video With Deep Neural Embeddings (2019, methodology): “However, recent advances in learning with deep visual embeddings have begun to produce unsupervised representations that rival the visual task transfer power of representations learned by their supervised counterparts [37, 36, 40, 3].”
Furnishing Your Room by What You See: An End-to-End Furniture Set Retrieval Framework with Rich Annotated Benchmark Dataset (2019, background): “Recent works on unsupervised learning, like DeepClustering ([6]), present the effectiveness of an iteration learning strategy between clustering and fine-tuning.”
iBOT: Image BERT Pre-Training with Online Tokenizer (2021, background): “In fact, semantics are now believed to emerge as long as a similarity between two distorted views of one image is enforced (Grill et al., 2020; He et al., 2020; Caron et al., 2020; 2018).”
MARE: Self-Supervised Multi-Attention REsu-Net for Semantic Segmentation in Remote Sensing (2021, neutral): “, DeepCluster [55]) approaches have been proposed.”