Efficient Visual Pretraining with Contrastive Detection

By Olivier J. H'enaff, Skanda Koppula, Jean-Baptiste Alayrac, Aäron van den Oord, Oriol Vinyals, João Carreira. 2021

190 citations recorded by the source corpus.

Self-supervised pretraining has been shown to yield powerful representations for transfer learning. These performance gains come at a large computational cost however, with state-of-the-art methods requiring an order of magnitude more computation than supervised pretraining. We tackle this computational bottleneck by introducing a new self-supervised objective, contrastive detection, which tasks representations with identifying object-level features across augmentations. This objective extracts a rich learning signal per image, leading to state-of-the-art transfer accuracy on a variety of downstream tasks, while requiring up to 10× less pretraining. In particular, our strongest ImageNet-pretrained model performs on par with SEER, one of the largest self-supervised systems to date, which uses 1000× more pretraining data. Finally, our objective seamlessly handles pretraining on more complex images such as those in COCO, closing the gap with supervised transfer learning from COCO to PASCAL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yue Cao
2. Zheng Zhang
3. Han Hu
4. Zhenda Xie
5. Yutong Lin
6. Jianfeng Gao
7. Yonglong Tian
8. Zhuliang Yao

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Xiaogang Wang
4. Ming-Hsuan Yang
5. Alan L. Yuille
6. Trevor Darrell
7. Jitendra Malik
8. Andrew Zisserman

In Representation & Self-Supervised Learning

1. Yonglong Tian
2. Olivier J. H'enaff
3. Aäron van den Oord
4. Zheng Zhang
5. Yue Cao
6. Han Hu
7. Zhenda Xie
8. Yutong Lin

Citations in the corpus

• Divide and Contrast: Self-supervised Learning from Uncurated Data (2021, background): “Such views can be created by color decomposition [88], patch cropping [69, 43, 3], data augmentation [12, 13, 83], or image segmentation [42, 92, 109].”
• Self-Supervised Learning with Swin Transformers (2021, methodology): “While we note that previous SSL works using ResNet as the backbone architecture usually report stronger performance over the supervised methods [9, 19, 12], no gains over supervised methods are observed using Transformer architectures.”