When Does Self-supervision Improve Few-shot Learning?

By Jong-Chyi Su, Subhransu Maji, Bharath Hariharan. 2019

190 citations recorded by the source corpus.

We investigate the role of self-supervised learning (SSL) in the context of few-shot learning. Although recent research has shown the benefits of SSL on large unlabeled datasets, its utility on small datasets is relatively unexplored. We find that SSL reduces the relative error rate of few-shot meta-learners by 4%-27%, even when the datasets are small and only utilizing images within the datasets. The improvements are greater when the training set is smaller or the task is more challenging. Although the benefits of SSL may increase with larger training sets, we observe that SSL can hurt the performance when the distributions of images used for meta-learning and SSL are different. We conduct a systematic study by varying the degree of domain shift and analyzing the performance of several meta-learners on a multitude of domains. Based on this analysis we present a technique that automatically selects images for SSL from a large, generic pool of unlabeled images for a given dataset that provides further improvements.

Subfields: Artificial Intelligence, Cancer Research, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cancer Research

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

CrossTransformers: spatially-aware few-shot transfer (2020, background): “rch, which inherently deals with transfer from pretext tasks to semantic ones and must therefore represent more than their training data [2, 11, 13, 18, 21, 27, 33, 44, 56, 92, 93]. Some recent works [26, 73] demonstrate that this can improve few-shot learning, although these use self-supervised aux”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, methodology): “Similarly, The pre-tasks were also used to color the image [128] and predict the relative position of each patch [129, 130], local Fisher discriminant [131].”
Task Agnostic Representation Consolidation: a Self-supervised based Continual Learning Approach (2022, result): “…than supervised pre-training in CL. Owing to additional computational effort, some of the approaches, e.g. (Zhang et al., 2020; Caccia et al., 2021; Mazumder et al., 2021; Su et al., 2020), relinquished pre-training altogether and employed auxiliary pretext task to boost task-agnostic learning.”
DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning (2020, background): “classification problem, such as works based on graph theories [66], [67], [68], [69], reinforcement learning [70], differentiable SVM [71], generative models [72], [73], [74], [75], [76], [77], [78], [79], [80], transductive learning [81], [82], [83], [84], [85], recurrent models [86], [87], self-su”
Learning Representations by Predicting Bags of Visual Words (2020, background): “Moreover, recent work has shown that self-supervision can be beneficial to many other learning problems [10, 18, 28, 29, 58, 71], such as few-shot [18, 58] and semi-supervised [28, 71] learning, or training generative adversarial networks [10].”
Fine-Grained Image Analysis With Deep Learning: A Survey (2021, methodology): “Thus, it is desirable to develop ﬁne-grained few-shot learning (FGFS) methods [148], [198], [199], [200].”