Exploring Simple Siamese Representation Learning

By Xinlei Chen, Kaiming He. 2020

5,118 citations recorded by the source corpus.

Siamese networks have become a common structure in various recent models for unsupervised visual representation learning. These models maximize the similarity between two augmentations of one image, subject to certain conditions for avoiding collapsing solutions. In this paper, we report surprising empirical results that simple Siamese networks can learn meaningful representations even using none of the following: (i) negative sample pairs, (ii) large batches, (iii) momentum encoders. Our experiments show that collapsing solutions do exist for the loss and structure, but a stop-gradient operation plays an essential role in preventing collapsing. We provide a hypothesis on the implication of stop-gradient, and further show proof-of-concept experiments verifying it. Our "SimSiam" method achieves competitive results on ImageNet and downstream tasks. We hope this simple baseline will motivate people to rethink the roles of Siamese architectures for unsupervised representation learning. Code is made available.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Masked Autoencoders Are Scalable Vision Learners (2021, background): “Recently, contrastive learning [3, 22] has been popular, e.g ., [62, 43, 23, 7], which models image similarity and dissimilarity (or only similarity [21, 8]) between two or more views.”
Emerging Properties in Self-Supervised Vision Transformers (2021, neutral): “We do not use a predictor [23, 14], resulting in the exact same architecture in both student and teacher networks.”
An Empirical Study of Training Self-Supervised Vision Transformers (2021, methodology): “We adopt a symmetrized loss [17, 7, 12]: ctr(q1, k2)+ctr(q2, k1).”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Other non-contrastive methods exist, such as BYOL [35] and SimSiam [17], but they usually require additional care to prevent the representation network from collapsing.”
High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, background): “The latest, most successful, modern family of SSL approaches for images (Misra & Maaten, 2020; Chen et al., 2020; Chen & He, 2020; He et al., 2020; Grill et al., 2020; Caron et al., 2020; 2021; Zbontar et al., 2021; Bardes et al., 2021), have two noteworthy characteristics that markedly distinguish…”
CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2021, methodology): “CLIP as a contrastive learning method has two simultaneous goals, namely (i) increasing the similarity of matched language-image pairs and (ii) decreasing the similarity of unmatched language-image pairs.”
ScaleNet: An Unsupervised Representation Learning Method for Limited Information (2023, methodology): “MoCo [20], SimCLR [6], BYOL [18], SIMSIAM [8] as contrastive learning methods, modify a two-branch network architectures to generate two different augmentations of an image and maximize the similarity outputs from two branches.”
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, background): “sentations are often learned through instance classification, in which each image and its augmentations are treated as a single output class to be pulled together [14], [15] or contrasted against other negative samples [22].”
Unified Contrastive Learning in Image-Text-Label Space (2022, background): “This augmented-view-based paradigm has also been extended to non-contrastive methods [4,7,20,37], where only positive image view pairs are considered in learning.”
How to Represent Part-Whole Hierarchies in a Neural Network (2021, background): “Contrastive learning has been applied very successfully to learn representations of image crops [Chen et al., 2020a, Bachman et al., 2019, He et al., 2020, Chen et al., 2020b, Tejankar et al., 2020] It learns to make the representations of two diﬀerent crops of the same image agree and the…”
Towards Label-Efficient Incremental Learning: A Survey (2023, methodology): “Buffer Self-Supervised Learning [Purush-walkam et al. , 2022] is a memory-based approach that extends an off-the-shelf self-supervision algorithm, Sim-Siam [Chen and He, 2021] for the case of incremental learning.”
Class Gradient Projection For Continual Learning (2022, background): “These studies [9, 10] have resolved practical limitations that have previously made learning difficult such as negative sample pairs, large batch size, and momentum encoders.”
Few-Shot Class-Incremental Learning from an Open-Set Perspective (2022, positive): “Inspired by self-supervised learning [2,3], we use two augmentations of each image to enhance training data diversity.”
CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One (2022, positive): “SimSiam [8] Unsupervised Continual Learning (UCL): has the unique property of being naturally robust to catastrophic forgetting when fine-tuning on incremental unlabeled data [16,25].”
The Challenges of Continuous Self-Supervised Learning (2022, methodology): “On each stream, we train the baseline SimSiam (Conventional SSL), SimSiam augmented with replay buffers (Buffered SSL) and SimSiam augmented with MinRed buffers (Buffered SSL (MinRed)).”
Self-Supervised Models are Continual Learners (2021, positive): “However, one more thing that is worth mentioning is that the success of the predictor network might also be related to the findings in SimSiam [9], BYOL [13] and DirectPred [25].”
Representational Continuity for Unsupervised Continual Learning (2021, methodology): “Chen & He (2021) tackled these limitations and proposed SimSiam , that use standard Siamese networks (Bromley et al., 1994) with the stop-gradient operation to prevent the collapsing of Siamese networks to a constant.”
Co2L: Contrastive Continual Learning (2021, background): “These core concepts have been followed by studies [10, 20, 15, 12] that have resolved practical limitations that have previously made learning difﬁcult such as negative sample pairs, large batch size, and momentum encoders.”
Pre-Trained Models: Past, Present and Future (2021, background): “ing and Transformers in NLP, some researchers explore self-supervised learning (Wu et al., 2018; Chen et al., 2020c; Chen and He, 2020; He et al., 2020) and Transformers (Carion et al.”