High Fidelity Visualization of What Your Self-Supervised Representation Knows About
By Florian Bordes, Randall Balestriero, Pascal Vincent. 2021
79 citations recorded by the source corpus.
Discovering what is learned by neural networks remains a challenge. In self-supervised learning, classification is the most common task used to evaluate how good a representation is. However, relying only on such downstream task can limit our understanding of what information is retained in the representation of a given input. In this work, we showcase the use of a Representation Conditional Diffusion Model (RCDM) to visualize in data space the representations learned by self-supervised models. The use of RCDM is motivated by its ability to generate high-quality samples -- on par with state-of-the-art generative models -- while ensuring that the representations of those samples are faithful i.e. close to the one used for conditioning. By using RCDM to analyze self-supervised models, we are able to clearly show visually that i) SSL (backbone) representation are not invariant to the data augmentations they were trained with -- thus debunking an often restated but mistaken belief; ii) SSL post-projector embeddings appear indeed invariant to these data augmentation, along with many other data symmetries; iii) SSL representations appear more robust to small adversarial perturbation of their inputs than representations trained in a supervised manner; and iv) that SSL-trained representations exhibit an inherent structure that can be explored thanks to RCDM visualization and enables image manipulation.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Representation & Self-Supervised Learning, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Yann LeCun
- Ruslan Salakhutdinov
- Mohit Bansal
- Jianfeng Gao
- Armand Joulin
- Ishan Misra
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Jitendra Malik
- Trevor Darrell
- Xiaogang Wang
- Andrea Vedaldi
- Andrew Zisserman
In Statistical and Nonlinear Physics
- Mark Chen
- Alex Nichol
- Prafulla Dhariwal
- Pascal Vincent
- Florian Bordes
- Randall Balestriero
- Tom Goldstein
- Micah Goldblum
In Representation & Self-Supervised Learning
- Ishan Misra
- Yann LeCun
- Armand Joulin
- Piotr Bojanowski
- Pascal Vincent
- Mathilde Caron
- Nicolas Ballas
- Yoshua Bengio
In Safety, Robustness & Interpretability
- Avi Schwarzschild
- Micah Goldblum
- Pascal Vincent
- Andrew Gordon Wilson
- Yann LeCun
- Ishan Misra
- Tom Goldstein
- Randall Balestriero
Citations in the corpus
- Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, background): “[3] train diffusion models conditioned on image representations from contrastive models.”
- A Cookbook of Self-Supervised Learning (2023, background): “It is worth noting that perfect invariance is not achieved thanks to the projector [Bordes et al., 2022a], which helps improve performance on tasks which are not entirely invariant.”
- Improving Visual Prompt Tuning for Self-supervised Vision Transformers (2023, background): “Depending on the pretraining strategy, the pretrained neural networks encode information differently in terms of the amount and content (Zhao et al., 2020; Bordes et al., 2021).”
- Masked Siamese Networks for Label-Efficient Learning (2022, methodology): “We follow the RCDM framework (Bordes et al., 2021) and train a conditional generative diffusion model, which maps a learned image representation back to pixel space.”