Distilling Localization for Self-Supervised Representation Learning
By Nanxuan Zhao, Zhirong Wu, Rynson W. H. Lau, Stephen Lin. 2020
59 citations recorded by the source corpus.
Recent progress in contrastive learning has revolutionized unsupervised representation learning. Concretely, multiple views (augmentations) from the same image are encouraged to map to close embeddings, while views from different images are pulled apart.In this paper, through visualizing and diagnosing classification errors, we observe that current contrastive models are ineffective at localizing the foreground object, limiting their ability to extract discriminative high-level features. This is due to the fact that view generation process considers pixels in an image uniformly.To address this problem, we propose a data-driven approach for learning invariance to backgrounds. It first estimates foreground saliency in images and then creates augmentations by copy-and-pasting the foreground onto a variety of back-grounds. The learning still follows an instance discrimination approach, so that the representation is trained to disregard background content and focus on the foreground. We study a variety of saliency estimation methods, and find that most methods lead to improvements for contrastive learning. With this approach, significant performance is achieved for self-supervised learning on ImageNet classification, and also for object detection on PASCAL VOC and MSCOCO.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Neil Houlsby
- Xiaohua Zhai
- Quoc V. Le
- Cordelia Schmid
- Dacheng Tao
- Jianfeng Gao
- Yoshua Bengio
- Lucas Beyer
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Jitendra Malik
- Xiaogang Wang
- Ming-Hsuan Yang
- Cordelia Schmid
- Andrew Zisserman
In Representation & Self-Supervised Learning
- Neil Houlsby
- Phillip Isola
- Xiaohua Zhai
- Yonglong Tian
- Lucas Beyer
- Ishan Misra
- Alexander Kolesnikov
- Michael Tschannen
Citations in the corpus
- On Robustness and Transferability of Convolutional Neural Networks (2020, methodology): “Creating synthetic datasets by inserting objects onto backgrounds has been used for training [75, 16, 21] and evaluating models [36], but previous works do not systematically vary object size, location or orientation, or analyze translation and rotation robustness only at the image level [18].”
- What makes for good views for contrastive learning (2020, background): “For example: luminance and chrominance decomposition [53], randomly augmenting an image twice [58, 7, 4, 23, 60, 50, 65, 67], using different time-steps of videos [42, 66, 46, 22, 21], patches of the same image [27, 42, 26], multiple sensory data [39, 9, 44], or text and its context [37, 59, 35, 32]”
- A Cookbook of Self-Supervised Learning (2023, background): “…learning algorithms, express seemingly contradictory findings, namely that existing self-supervised learning strategies are or are not effective for downstream dense prediction tasks [Goyal et al., 2019, Purushwalkam and Gupta, 2020, Zhao et al., 2021, Ericsson et al., 2021b, Shwartz-Ziv et al.].”