Small Data Challenges in Big Data Era: A Survey of Recent Progress on Unsupervised and Semi-Supervised Methods

By Guo-Jun Qi, Jiebo Luo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019

300 citations recorded by the source corpus.

Representation learning with small labeled data have emerged in many problems, since the success of deep neural networks often relies on the availability of a huge amount of labeled data that is expensive to collect. To address it, many efforts have been made on training sophisticated models with few labeled data in an unsupervised and semi-supervised fashion. In this paper, we will review the recent progresses on these two major categories of methods. A wide spectrum of models will be categorized in a big picture, where we will show how they interplay with each other to motivate explorations of new ideas. We will review the principles of learning the transformation equivariant, disentangled, self-supervised and semi-supervised representations, all of which underpin the foundation of recent progresses. Many implementations of unsupervised and semi-supervised generative models have been developed on the basis of these criteria, greatly expanding the territory of existing autoencoders, generative adversarial nets (GANs) and other deep networks by exploring the distribution of unlabeled data for more powerful representations. We will discuss emerging topics by revealing the intrinsic connections between unsupervised and semi-supervised learning, and propose in future directions to bridge the algorithmic and theoretical gap between transformation equivariance for unsupervised learning and supervised invariance for supervised learning, and unify unsupervised pretraining and supervised finetuning. We will also provide a broader outlook of future directions to unify transformation and instance equivariances for representation learning, connect unsupervised and semi-supervised augmentations, and explore the role of the self-supervised regularization for many learning problems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/64904e0a9ee10d513836e5fe0a875fbdd8b92d79

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, background): “Our survey is related to other surveys on semi-supervised learning [8], [13], [17], self-supervised learning [18], or both topics [19].”
Image-to-Image Translation: Methods and Applications (2021, methodology): “Therefore, we refer to the universal taxonomy in machine learning, such as the categorizations used in [99], [100], [101],”
Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation (2021, background): “A surge of research interest has been recently attracted to unsupervised learning [42].”
A Survey on Deep Semi-Supervised Learning (2021, background): “Besides, [5] focuses on unifying the evaluation indices of some consistency regularization methods, and [21] only”