Improvements to Context Based Self-Supervised Learning

By T. Mundhenk, Daniel E. Ho, Barry Y. Chen. 2017

125 citations recorded by the source corpus.

We develop a set of methods to improve on the results of self-supervised learning using context. We start with a baseline of patch based arrangement context learning and go from there. Our methods address some overt problems such as chromatic aberration as well as other potential problems such as spatial skew and mid-level feature neglect. We prevent problems with testing generalization on common self-supervised benchmark tests by using different datasets during our development. The results of our methods combined yield top scores on all standard self-supervised benchmarks, including classification and detection on PASCAL VOC 2007, segmentation on PASCAL VOC 2012, and "linear tests" on the ImageNet and CSAIL Places datasets. We obtain an improvement over our baseline method of between 4.0 to 7.1 percentage points on transfer learning classification tests. We also show results on different standard network architectures to demonstrate generalization as well as portability. All data, models and programs are available at: https://gdo-datasci.llnl.gov/selfsupervised/.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Self-Supervised Learning of Pretext-Invariant Representations (2019, neutral): “Prior pretext tasks include image colorization [9, 30, 38, 39, 86, 87], orientation prediction [20], affine transform prediction [85], predicting contextual image patches [10], reordering image patches [5, 21, 53, 54, 56], counting visual primitives [55], or their combinations [11].”
A critical analysis of self-supervision, or what we can learn from a single image (2019, background): “…namely, in-painting (Pathak et al., 2016), patch context and jigsaw puzzles (Doersch et al., 2015; Noroozi & Favaro, 2016; Noroozi et al., 2018; Mundhenk et al., 2017), clustering (Caron et al., 2018), noise-as-targets (Bojanowski & Joulin, 2017), colorization (Zhang et al., 2016; Larsson et…”
Revisiting Self-Supervised Visual Representation Learning (2019, background): “These include a model from [31] that predicts the permutation of a “jigsaw puzzle” created from the full image and recent follow-ups [29, 33].”
Self-Supervised GANs via Auxiliary Rotation Loss (2018, background): “For example, the network can be trained to solve the context prediction problem, like the relative location of disjoint patches [24, 43] or the patch permutation of a jigsaw puzzle [44].”
Self-labelling via simultaneous clustering and representation learning (2019, background): “, 2016), patch context and jigsaw puzzles (Doersch et al., 2015; Noroozi & Favaro, 2016; Noroozi et al., 2018; Mundhenk et al., 2017), clustering (Caron et al.”
Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules (2019, background): “Augmentation has been shown to have a large impact on image modalities where data is scare or expensive to generate, like medical imaging (Bowles et al., 2018; Frid-Adar et al., 2018) or non-supervised learning approaches (Mundhenk et al., 2017).”
High-Fidelity Image Generation With Fewer Labels (2019, neutral): “Other forms of self-supervision include predicting relative locations of disjoint image patches of a given image (Doersch et al., 2015; Mundhenk et al., 2018) or estimating the permutation of randomly swapped image Figure 2.”
Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction. (2018, background): “Recently, a number of self-supervised learning methods have been proposed to for representation learning from images and videos [6, 7, 29, 32, 36, 38, 17].”
Cross Pixel Optical Flow Similarity for Self-Supervised Learning (2018, neutral): “so cluster features from a pre-trained network to generate pseudo labels, which allows for knowledge distillation from larger networks into smaller ones. The latest iteration on context prediction by [30] obtains state-of-the-art results on several benchmarks. Adversarial/Generative: BiGAN based pre”
Understanding Deep Learning Techniques for Image Segmentation (2019, background): “imultaneously learns features specic to parts of an object as well as their semantic relationships. These patch based self supervision techniques can be further improved using contextual information [152]. Using context-encoders [164] can also derive spatial and semantic relationship among various p”
Automatic Shortcut Removal for Self-Supervised Representation Learning (2020, background): “More recently, refined color augmentation heuristics such as chroma blurring have been developed for patch-based pretext tasks (Mundhenk et al., 2018).”
Efficient Visual Pretraining with Contrastive Detection (2021, background): “Early selfsupervised pretraining typically involved image restoration, including denoising [60], inpainting [46], colorization [66, 36], egomotion prediction [1], and more [15, 43, 67].”
Self-Supervised Learning in Remote Sensing: A review (2022, background): “There have been some early studies on the evaluation of different pretext tasks and data augmentations and possible improvements [250, 251], yet they mainly focus on natural images and evaluate on image-level classiﬁcation tasks.”
Sample-Efficient Deep Learning for COVID-19 Diagnosis Based on CT Scans (2020, background): “Examples of auxiliary tasks include rotation prediction [32], image inpainting [33], automatic colorization [34], context prediction [35], etc.”
Self-supervised Spatiotemporal Feature Learning by Video Geometric Transformations (2018, background): “Recently, many self-supervised learning methods were proposed to learn image representations [6, 7, 23, 26, 31, 34].”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, methodology): “3) Generative Algorithms: For the category of generative algorithms, this study primarily focuses on MIM methods.”
Are we pretraining it right? Digging deeper into visio-linguistic pretraining (2020, background): “This work spawned multiple works around predicting the “jigsaw puzzle” [35,36].”
MARE: Self-Supervised Multi-Attention REsu-Net for Semantic Segmentation in Remote Sensing (2021, background): “The strategies were several, such as patch context [40,41], in-painting [42], colorization [43,44], jigsaw puzzles [45], noise [46], generation [47], rotation [16]), and counting [48].”
Self-Supervised Learning via Conditional Motion Propagation (2019, unknown): “[24] Context 40.”