Self-Supervised Learning via Conditional Motion Propagation

By Xiaohang Zhan, Xingang Pan, Ziwei Liu, Dahua Lin, Chen Change Loy. 2019

53 citations recorded by the source corpus.

Intelligent agent naturally learns from motion. Various self-supervised algorithms have leveraged the motion cues to learn effective visual representations. The hurdle here is that motion is both ambiguous and complex, rendering previous works either suffer from degraded learning efficacy, or resort to strong assumptions on object motions. In this work, we design a new learning-from-motion paradigm to bridge these gaps. Instead of explicitly modeling the motion probabilities, we design the pretext task as a conditional motion propagation problem. Given an input image and several sparse flow guidance on it, our framework seeks to recover the full-image motion. Compared to other alternatives, our framework has several appealing properties: (1) Using sparse flow guidance during training resolves the inherent motion ambiguity, and thus easing feature learning. (2) Solving the pretext task of conditional motion propagation encourages the emergence of kinematically-sound representations that poss greater expressive power. Extensive experiments demonstrate that our framework learns structural and coherent features; and achieves state-of-the-art self-supervision performance on several downstream tasks including semantic segmentation, instance segmentation and human parsing. Furthermore, our framework is successfully extended to several useful applications such as semi-automatic pixel-level annotation.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Self-Supervised Learning of Video-Induced Visual Invariances (2019, neutral): “The ambiguity in single-frame prediction has been side-stepped via time-agnostic prediction [29], motion segmentation [47], cross-pixel matching [38], and by giving the model a motion cue as input [70].”
Understanding Deep Learning Techniques for Image Segmentation (2019, background): “Other self supervision techniques can leverage motion cues in videos to segment various parts of an object for better semantic understanding [216].”
Steering Self-Supervised Feature Learning Beyond Local Pixel Statistics (2020, unknown): “[63] [63] - 44.”
Self-Supervised Visual Representation Learning from Hierarchical Grouping (2020, unknown): “Approaches to self-supervised visual learning that train networks to predict one aspect of the data from another have utilized a variety of proxy tasks, including prediction of context [38], colorization [50, 29], cross-channels [51], optical-ﬂow [49], and rotation [16].”
Knowledge Distillation Meets Self-Supervision (2020, methodology): “Based on pretext tasks, SS methods can be grouped into several categories, including constructionbased methods such as inpainting [35] and colorization [50], prediction-based methods [7,9,11,21,28,30,31,47,49], cluster-based methods [3,48], generation-based methods [8,10,12] and contrastive-based me”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, methodology): “Others : The pretext task employed in [44], [45] involved a conditional motion propagation problem.”