Self-Supervised Learning: Generative or Contrastive

By Xiao Liu, Fanjin Zhang, Zhenyu Hou, Zhaoyu Wang, Li Mian, Jing Zhang, Jie Tang. 2020

2,207 citations recorded by the source corpus.

Deep supervised learning has achieved great success in the last decade. However, its defects of heavy dependence on manual labels and vulnerability to attacks have driven people to find other paradigms. As an alternative, self-supervised learning (SSL) attracts many researchers for its soaring performance on representation learning in the last several years. Self-supervised representation learning leverages input data itself as supervision and benefits almost all types of downstream tasks. In this survey, we take a look into new self-supervised learning methods for representation in computer vision, natural language processing, and graph learning. We comprehensively review the existing empirical methods and summarize them into three main categories according to their objectives: generative, contrastive, and generative-contrastive (adversarial). We further collect related theoretical analysis on self-supervised learning to provide deeper thoughts on why self-supervised learning works. Finally, we briefly discuss open problems and future directions for self-supervised learning. An outline slide for the survey is provided<inline-formula><tex-math notation="LaTeX">$^1$</tex-math><alternatives><mml:math><mml:msup><mml:mrow/><mml:mn>1</mml:mn></mml:msup></mml:math><inline-graphic xlink:href="liu-ieq1-3090866.gif"/></alternatives></inline-formula>.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

CogView: Mastering Text-to-Image Generation via Transformers (2021, background): “(Paul Cézanne [17]) As contrastive self-supervised pretraining has revolutionized computer vision (CV) [24, 21, 8, 32], visual-language pretraining, which brings high-level semantics to images, is becoming the next frontier of visual understanding [38, 30, 39].”
GPT Understands, Too (2021, methodology): “Self-supervised (Liu et al., 2020) pre-trained language models (Han et al., 2021) including GPT (Rad-ford et al., 2019), BERT (Devlin et al., 2018), XL-Net (Yang et al., 2019), RoBERTa (Liu et al., 2019) have been observed to learn not only contextualized text representations but also linguistic…”
GLM-130B: An Open Bilingual Pre-trained Model (2022, background): “Nevertheless, some recent attempts has been on parameter-efficient learning on language models (Houlsby et al., 2019) and prompt tuning (i.e., P-tuning, Li & Liang (2021); Liu et al. (2021b); Lester et al. (2021); Liu et al. (2022)).”
Continual Barlow Twins: Continual Self-Supervised Learning for Remote Sensing Semantic Segmentation (2022, background): “Recently, referring to CV in general, several innovations have been proposed [53], [54], reviewed in many surveys [24], [55], [56], [57], [58].”
Pre-Trained Models: Past, Present and Future (2021, methodology): “To take full advantage of largescale unlabeled corpora to provide versatile linguistic knowledge for NLP tasks, the NLP community adopts self-supervised learning (Liu et al., 2020b) to develop PTMs.”
P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks (2021, background): “Pre-trained Language Models Selfsupervised (Liu et al., 2020) pre-trained language models has become the backbone of natural language processing.”
Towards mental time travel: a hierarchical memory for reinforcement learning agents (2021, background): “Thus, as in prior work referenced above, we show that self-supervised learning [32], is necessary for successful learning on our tasks (App.”
How Well Do Self-Supervised Models Transfer? (2020, background): “Self-supervised learning Self-supervised representation learning is now a large topic that it is impossible to cover completely here, and we point the reader to excellent recent surveys [26, 37] for thorough reviews.”
Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning (2021, background): “Contrastive self-supervised learning is a signiﬁcant brunch of self-supervised learning [51].”
Masked Autoencoders for Point Cloud Self-supervised Learning (2022, background): “The main ideas can be summarized as: a) supervision labels are generated from the data itself instead of human annotating, b) the model predicts parts of the data from other parts [22].”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “The advancement of SSL [3], [4], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24] has exhibitedrapidprogress, capturing signiﬁcant attention within the research community (Fig.”
Deep Neural Networks and Tabular Data: A Survey (2021, neutral): “These methods have also shown astonishing results in computer vision and natural language processing [215], [216].”
Are Graph Augmentations Necessary?: Simple Graph Contrastive Learning for Recommendation (2021, background): “Recently, a resurgence of contrastive learning (CL) [11, 12, 16] has been witnessed in deep representation learning.”
Joint Learning of Localized Representations from Medical Images and Reports (2021, background): “in pixel-space) and therefore focus too much on low-level details (like pixels) instead of encouraging high-level abstractions that are typically required for discriminative downstream tasks like classification[46].”
Text Transformations in Contrastive Self-Supervised Learning: A Review (2022, methodology): “Based on the objective function of the deep neural networks, the self-supervised models can be divided into three major groups: generative, contrastive, and generativecontrastive (or adversarial) [Liu et al., 2021].”
Self-Supervised Learning of Graph Neural Networks: A Unified Review (2021, methodology): “While both [62] and our work review SSL methods as contrastive ones and non-contrastive ones, we distinguish our reviews from [62] by the following distinct differences.”
OAG-BERT: Pre-train Heterogeneous Entity-augmented Academic Language Models (2021, methodology): “Our proposed OAG-BERT model is based on BERT [9], a selfsupervised [29] bidirectional language model.”