Automatic Shortcut Removal for Self-Supervised Representation Learning
By Matthias Minderer, Olivier Bachem, Neil Houlsby, Michael Tschannen. 2020
79 citations recorded by the source corpus.
In self-supervised visual representation learning, a feature extractor is trained on a "pretext task" for which labels can be generated cheaply, without human annotation. A central challenge in this approach is that the feature extractor quickly learns to exploit low-level visual features such as color aberrations or watermarks and then fails to learn useful semantic representations. Much work has gone into identifying such "shortcut" features and hand-designing schemes to reduce their effect. Here, we propose a general framework for mitigating the effect shortcut features. Our key assumption is that those features which are the first to be exploited for solving the pretext task may also be the most vulnerable to an adversary trained to make the task harder. We show that this assumption holds across common pretext tasks and datasets by training a "lens" network to make small image changes that maximally reduce performance in the pretext task. Representations learned with the modified images outperform those learned without in all tested cases. Additionally, the modifications made by the lens reveal how the choice of pretext task and dataset affects the features learned by self-supervision.
Subfields: Artificial Intelligence, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Cordelia Schmid
- Richard S. Zemel
- Yoshua Bengio
- Chen Sun
- Matthias Bethge
- Bernt Schiele
- Wieland Brendel
- Sergey Levine
In Representation & Self-Supervised Learning
- Yonglong Tian
- Phillip Isola
- Dilip Krishnan
- Cordelia Schmid
- Matthias Bethge
- Wieland Brendel
- Chen Sun
- Claudio Michaelis
Citations in the corpus
- Shortcut learning in deep neural networks (2020, background): “For example, transferring model performance across datasets (domain transfer) is challenging because models often use domain-specific shortcut features, and shortcuts limit the usefulness of unsupervised representations [54].”
- What makes for good views for contrastive learning (2020, background): “In practice, “shortcuts” are one type of task-irrelevant nui-sance and should be removed from the views [45].”
- Patch SVDD: Patch-level SVDD for Anomaly Detection and Segmentation (2020, methodology): “In addition, the results in previous studies [6,7] show that the features of a randomly initialized encoder might be used to distinguish anomalies.”
- Geographical Knowledge-Driven Representation Learning for Remote Sensing Images (2021, background): “Each class is represented by a value among [10, 20, 30, 40, 50, 60, 70, 80, 90, 100].”
- Text Data Augmentation for Deep Learning (2021, methodology): “Minderer et al. [54] use this technique to facilitate self-supervised pretext tasks.”
- Unsupervised Contrastive Learning of Sound Event Representations (2020, background): “These undesired patterns are denoted as shortcuts in computer vision [30].”