Cross Pixel Optical Flow Similarity for Self-Supervised Learning

By Aravindh Mahendran, James Thewlis, Andrea Vedaldi. 2018

68 citations recorded by the source corpus.

We propose a novel method for learning convolutional neural image representations without manual supervision. We use motion cues in the form of optical flow, to supervise representations of static images. The obvious approach of training a network to predict flow from a single image can be needlessly difficult due to intrinsic ambiguities in this prediction task. We instead propose a much simpler learning goal: embed pixels such that the similarity between their embeddings matches that between their optical flow vectors. At test time, the learned deep network can be used without access to video or flow information and transferred to tasks such as image classification, detection, and segmentation. Our method, which significantly simplifies previous attempts at using motion for self-supervision, achieves state-of-the-art results in self-supervision using motion cues, competitive results for self-supervision in general, and is overall state of the art in self-supervised pretraining for semantic image segmentation, as demonstrated on standard benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Self-Supervised Learning of Video-Induced Visual Invariances (2019, neutral): “The ambiguity in single-frame prediction has been side-stepped via time-agnostic prediction [29], motion segmentation [47], cross-pixel matching [38], and by giving the model a motion cue as input [70].”
Learning Visual Representations with Caption Annotations (2020, background): “Consequently, some works [7,24,42] have evaluated unsupervised methods trained on uncurated data [58].”
A critical analysis of self-supervision, or what we can learn from a single image (2019, background): “…Gao et al., 2016; Wang et al., 2017), foreground-background segmentation via video segmentation (Pathak et al., 2017), optical-ﬂow (Gan et al., 2018; Mahendran et al., 2018), future-frame synthesis (Srivastava et al., 2015), audio prediction from video (de Sa, 1994; Owens et al., 2016),…”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, neutral): “Many self-supervised methods manipulate the input data to extract a supervised signal in the form of a pretext task [1, 13, 26, 28, 30, 36, 38, 41, 42, 47, 48, 57].”
Self-labelling via simultaneous clustering and representation learning (2019, background): “, 2017), optical-flow (Gan et al., 2018; Mahendran et al., 2018), futureframe synthesis (Srivastava et al.”
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey (2019, methodology): “Mehtod SubCategory Code Contribution VideoGAN [85] Generation 3 Forerunner of video generation with GAN MocoGAN [86] Generation 3 Decomposing motion and content for video generation with GAN TemporalGAN [144] Generation 3 Decomposing temporal and image generator for video generation Video Colorizati”
Self-Supervised Visual Representation Learning from Hierarchical Grouping (2020, unknown): “[33] learn pixel-wise embeddings for static frames, such that their pair-wise similarity mirrors that of optical flow vectors.”
Leveraging Large-Scale Uncurated Data for Unsupervised Pre-training of Visual Features (2019, background): “Many pretext tasks have been proposed [22, 31, 47, 51], exploiting, amongst others, spatial context [12, 24, 33, 34, 36], cross-channel prediction [27, 28, 55, 56], or the temporal structure of videos [1, 35, 46].”
Self-Supervised Learning via Conditional Motion Propagation (2019, background): “[20] assumed that pixels with similar features should have similar motions, and designed a cross pixel flow similarity loss to optimize the representations.”
Conditional Object-Centric Learning from Video (2021, methodology): “The self-supervision proxy task was that of single image optical flow prediction, inspired by prior work on predicting optical flow from static images (Mahendran et al., 2019; Walker et al., 2015).”
Joint Learning of Localized Representations from Medical Images and Reports (2021, background): “Therefore, there is a number of recent approaches that use region-level contrast[87,86,83,6,61,52], i.”
Unsupervised Pre-Training of Image Features on Non-Curated Data (2019, unknown): “Many pretext tasks have been proposed [22, 31, 47, 51], exploiting, amongst others, spatial context [12, 24, 33, 34, 36], cross-channel prediction [27, 28, 55, 56], or the temporal structure of videos [1, 35, 46].”