Do Self-Supervised and Supervised Methods Learn Similar Visual Representations?

By Tom George Grigg, Dan Busbridge, Jason Ramapuram, Russ Webb. 2021

31 citations recorded by the source corpus.

Despite the success of a number of recent techniques for visual self-supervised deep learning, there has been limited investigation into the representations that are ultimately learned. By leveraging recent advances in the comparison of neural representations, we explore in this direction by comparing a contrastive self-supervised algorithm to supervision for simple image data in a common architecture. We find that the methods learn similar intermediate representations through dissimilar means, and that the representations diverge rapidly in the final few layers. We investigate this divergence, finding that these layers strongly fit to their distinct learning objectives. We also find that the contrastive objective implicitly fits the supervised objective in intermediate layers, but that the reverse is not true. Our work particularly highlights the importance of the learned intermediate representations, and raises critical questions for auxiliary task design.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Yoshua Bengio
2. Quoc V. Le
3. Dong Yu
4. Abdel-rahman Mohamed
5. Jinyu Li
6. Mohit Bansal
7. Hung-yi Lee
8. Dacheng Tao

Citations in the corpus

• StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, neutral): “similar [Grigg et al. 2021], whereas ViTs and CNNs learn different representations [Raghu et al.”
• No One Representation to Rule Them All: Overlapping Features of Training Methods (2021, background): “Concurrent work similarly finds that different training objectives yield different final-layer feature embeddings (Grigg et al., 2021).”
• Self-Supervised Speech Representation Learning: A Review (2022, background): “Interestingly, similar network behavior was also recently described for self-supervised learners in computer vision: using a contrastive self-supervised learner (SimCLR) that optimizes for augmentation invariance, [270] show that it is the intermediate representations that most closely approximate i”