Self-Supervised Feature Learning by Learning to Spot Artifacts

By Simon Jenni, Paolo Favaro. 2018

133 citations recorded by the source corpus.

We introduce a novel self-supervised learning method based on adversarial training. Our objective is to train a discriminator network to distinguish real images from images with synthetic artifacts, and then to extract features from its intermediate layers that can be transferred to other data domains and tasks. To generate images with artifacts, we pre-train a high-capacity autoencoder and then we use a damage and repair strategy: First, we freeze the autoencoder and damage the output of the encoder by randomly dropping its entries. Second, we augment the decoder with a repair network, and train it in an adversarial manner against the discriminator. The repair network helps generate more realistic images by inpainting the dropped feature entries. To make the discriminator focus on the artifacts, we also make it predict what entries in the feature were dropped. We demonstrate experimentally that features learned by creating and spotting artifacts achieve state of the art performance in several benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Visual Representations with Caption Annotations (2020, background): “tasks consist in predicting missing pieces in manipulated images, for instance context prediction [15], colorization [13,36,66], inpainting of missing portions [46], prediction of image rotations [19], spotting artifacts [31], or cross-channel prediction [67].”
A critical analysis of self-supervision, or what we can learn from a single image (2019, neutral): “…clustering (Caron et al., 2018), noise-as-targets (Bojanowski & Joulin, 2017), colorization (Zhang et al., 2016; Larsson et al., 2017), generation (Jenni & Favaro, 2018; Ren & Lee, 2018; Donahue et al., 2017), geometry (Dosovitskiy et al., 2016; Gidaris et al., 2018) and counting (Noroozi et al.,…”
Local Aggregation for Unsupervised Learning of Visual Embeddings (2019, methodology): “Examples include context prediction [14], colorization of grayscale images [14], inpainting of missing portions of images [56], predicting rotations of images [20], spotting artifacts of images [36], and the SplitBrain method [76].”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, neutral): “Many self-supervised methods manipulate the input data to extract a supervised signal in the form of a pretext task [1, 13, 26, 28, 30, 36, 38, 41, 42, 47, 48, 57].”
Self-labelling via simultaneous clustering and representation learning (2019, neutral): “…al., 2018; Huang et al., 2019), noise-as-targets (Bojanowski & Joulin, 2017), colorization (Zhang et al., 2016; Larsson et al., 2017), generation (Jenni & Favaro, 2018; Ren & Lee, 2018; Donahue et al., 2017; Donahue & Simonyan, 2019), predictive coding (Oord et al., 2018; Hénaff et al., 2019),…”
Cross Pixel Optical Flow Similarity for Self-Supervised Learning (2018, neutral): “Motion Ours direct reg. Optical-Flow Self 14.0 22.6 25.3 25.0 23.0 es Doersch et.al. [22] Context [26] 16.2 23.3 30.2 31.7 29.6 Gidaris et.al. [37] Rotation [37] 18.8 31.7 38.7 38.2 36.5 Jenni et.al. [32] - [32] 19.5 33.3 37.9 38.9 34.9 Mundhenk et.al. [30] Context [30] 19.6 31.4 37.0 37.8 33.3 Noro”
Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision (2022, background): “Alternatively, many creative pretext tasks have also been proposed, showing that good features can be trained that way [1, 34, 65, 70, 77, 85, 89, 90, 96, 97, 124, 125, 143].”
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey (2019, unknown): “TABLE 2 Summary of Self-Supervised Image Feature Learning Methods Based on the Category of Pretext Tasks Method Category Code Contribution AutoColor [92] Generation Yes Training ConvNet to predict per-pixel color histograms BiGAN [93] Generation Yes Bidirectional GAN to project data into latent spac”
Self-Supervised Learning in Remote Sensing: A review (2022, methodology): “Jenni et al. [68] proposed a self-supervised learning method based on spotting and predicting artifacts in an adversarial manner.”
Unsupervised Pre-Training of Image Features on Non-Curated Data (2019, unknown): “Many pretext tasks have been proposed [22, 31, 44, 48], exploiting, amongst others, spatial context [12, 24, 33, 34, 36], cross-channel prediction [27, 28, 52, 53], or the temporal structure of videos [1, 35, 43].”
When Does Self-supervision Improve Few-shot Learning? (2019, background): “…(Noroozi et al., 2017), missing patches (Pathak et al., 2016; Trinh et al., 2019), motion segmentation labels (Pathak et al., 2017), artifacts (Jenni & Favaro, 2018), next patch feature (Oord et al., 2018; H´enaff et al., 2019), or by maximizing mutual information (Hjelm et al., 2019; Bachman…”
Steering Self-Supervised Feature Learning Beyond Local Pixel Statistics (2020, methodology): “Noroozi and Favaro [38] extend this idea to multiple patches by solving jigsaw puzzles.”
Learning To Classify Images Without Labels (2020, background): “Numerous pretext tasks have been explored in the literature, including predicting the patch context [13,33], inpainting patches [39], solving jigsaw puzzles [35,37], colorizing images [58,29], using adversarial training [14,15], predicting noise [3], counting [36], predicting rotations [16], spottin”
Self-Supervised Representation Learning by Rotation Feature Decoupling (2019, methodology): “Methods relying on adversarial training include [12] and [21].”
MixCo: Mix-up Contrastive Learning for Visual Representation (2020, methodology): “Contrastive Learning Although earlier self-supervised methods handcrafted pretext tasks [1, 10, 17, 20, 22, 33], most recent approaches that show remarkable performance are based on instance-level classification with contrastive loss [5, 6, 7, 12, 16, 26, 31].”
Geographical Knowledge-Driven Representation Learning for Remote Sensing Images (2021, background): “For example, occlude an area in the image randomly and repair it with networks, from which a good image representation is learned [17, 18, 35, 40].”
Exploring Balanced Feature Spaces for Representation Learning (2021, background): “A wide range of pretext tasks have been developed including image inpainting (Jenni & Favaro, 2018; Pathak et al., 2016), image colorization (Larsson et al., 2016; 2017), context prediction (Doersch et al., 2015), jigsaw puzzles (Carlucci et al., 2019; Noroozi & Favaro, 2016; Wei et al., 2019),…”
Self-Supervised Learning via Conditional Motion Propagation (2019, unknown): “[14] Spot-Artifacts 38.”
Multi-task Self-Supervised Learning for Human Activity Detection (2019, background): “Self-supervision has been actively studied recently in the vision domain, and several surrogate tasks have been proposed for learning representations from static images, videos, sound, and in robotics [3, 13, 15, 18, 19, 24, 30, 34, 42, 47, 51, 52, 54, 74].”