Masked Siamese Networks for Label-Efficient Learning
By Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael G. Rabbat, Nicolas Ballas. European Conference on Computer Vision, 2022
424 citations recorded by the source corpus.
We propose Masked Siamese Networks (MSN), a self-supervised learning framework for learning image representations. Our approach matches the representation of an image view containing randomly masked patches to the representation of the original unmasked image. This self-supervised pre-training strategy is particularly scalable when applied to Vision Transformers since only the unmasked patches are processed by the network. As a result, MSNs improve the scalability of joint-embedding architectures, while producing representations of a high semantic level that perform competitively on low-shot image classification. For instance, on ImageNet-1K, with only 5,000 annotated images, our base MSN model achieves 72.4% top-1 accuracy, and with 1% of ImageNet-1K labels, we achieve 75.7% top-1 accuracy, setting a new state-of-the-art for self-supervised learning on this benchmark. Our code is publicly available.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Media Technology
View Paper: https://www.semanticscholar.org/paper/a6e231db70a6774abc995afc0201671e0dab4d3a
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Armand Joulin
- Jianfeng Gao
- Ishan Misra
- Mohit Bansal
- Yann LeCun
- Quoc V. Le
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Jitendra Malik
- Xiaogang Wang
- Trevor Darrell
- Ming-Hsuan Yang
- Andrew Zisserman
- Andrea Vedaldi
In Media Technology
- Armand Joulin
- Ishan Misra
- Jean Ponce
- Julien Mairal
- Piotr Bojanowski
- Hamed Pirsiavash
- Yann LeCun
- Herv'e J'egou
Citations in the corpus
- Revisiting Feature Prediction for Learning Visual Representations from Video (2024, methodology): “…that perform well across many downstream tasks through linear probing or low-shot adaptation (Assran et al., 2023; Oquab et al., 2023; Assran et al., 2022), while demonstrating an efficiency gain during pretraining compared to pixel level reconstruction (Assran et al., 2023; Baevski…”
- A Cookbook of Self-Supervised Learning (2023, methodology): “To this effect some works have explored pretraining on large uncurated datasets [Goyal et al., 2021], or on datasets that different from ImageNet such as COCO [El-Nouby et al., 2021], or iNaturalist [Assran et al., 2022a].”
- Stabilizing Transformer Training by Preventing Attention Entropy Collapse (2023, methodology): “Self-Supervised Training of Visual Representations In computer vision, SSL has been effective in enabling efficient training on downstream tasks (Assran et al., 2022).”
- A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, methodology): “A similar teacher-student framework is adopted in MSN [27] and data2vec [24].”
- DINOv2: Learning Robust Visual Features without Supervision (2023, background): “We see the components proposed in this work lead to a very significant improvement (+4 .”