Towards Good Practices in Self-supervised Representation Learning
By Srikar Appalaraju, Yi Zhu, Yi Zhu, Yusheng Xie, Istv'an Feh'erv'ari. 2020
23 citations recorded by the source corpus.
Self-supervised representation learning has seen remarkable progress in the last few years. More recently, contrastive instance learning has shown impressive results compared to its supervised learning counterparts. However, even with the ever increased interest in contrastive instance learning, it is still largely unclear why these methods work so well. In this paper, we aim to unravel some of the mysteries behind their success, which are the good practices. Through an extensive empirical analysis, we hope to not only provide insights but also lay out a set of best practices that led to the success of recent work in self-supervised representation learning.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Florian Bordes
- Randall Balestriero
- Pascal Vincent
- Raghavan Manmatha
- Srikar Appalaraju
- Yusheng Xie
- Bhargava Urala Kota
- Bhavan A. Jasani
In Computer Vision and Pattern Recognition
- Florian Bordes
- Raghavan Manmatha
- Randall Balestriero
- Srikar Appalaraju
- Yusheng Xie
- Luc Van Gool
- Andrew Zisserman
- Cordelia Schmid
In Representation & Self-Supervised Learning
- Florian Bordes
- Randall Balestriero
- Pascal Vincent
- Raghavan Manmatha
- Srikar Appalaraju
- Yusheng Xie
- Bhavan A. Jasani
- Yoshua Bengio
Citations in the corpus
- High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, background): “Another possibility, explored in Zhao et al. (2021); Appalaraju et al. (2020); Ericsson et al. (2021), is to learn to invert the DN features through a Deep Image Prior (DIP) g θ as in min θ d ( g θ ( f ( x )) , x ) .”
- DocFormer: End-to-End Transformer for Document Understanding (2021, methodology): “Recent publications [6, 2] in self-supervision show that a deeper projection head with ReLU activation acts as a one-way filter to enrich the representation space.”