Cross and Learn: Cross-Modal Self-Supervision

By Nawid Sayed, Biagio Brattoli, Björn Ommer. 2018

86 citations recorded by the source corpus.

In this paper we present a self-supervised method for representation learning utilizing two different modalities. Based on the observation that cross-modal information has a high semantic meaning we propose a method to effectively exploit this signal. For our approach we utilize video data since it is available on a large scale and provides easily accessible modalities given by RGB and optical flow. We demonstrate state-of-the-art performance on highly contested action recognition datasets in the context of self-supervised learning. We show that our feature representation also transfers to other tasks and conduct extensive ablation studies to validate our core contributions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “As a result, and with the recent emergence of research video data sets [1, 59], videos have been investigated in the context of self-supervision (for example, [40, 64, 63, 28, 65, 73, 41, 51, 42, 3, 2]).”
S4L: Self-Supervised Semi-Supervised Learning (2019, background): “Self-supervised techniques have a variety of applications in a broad range of computer vision topics [14, 35, 7, 29, 34].”
Revisiting Self-Supervised Visual Representation Learning (2019, background): “Similarly, when learning representation from videos, one can either make use of the synchronized cross-modality stream of audio, video, and potentially subtitles [35, 39, 23, 44], or of the consistency in the temporal dimension [41].”
Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction. (2018, background): “…[28, 38], the context of images [25, 36], and the similarity of two patches from an image [6, 15, 33, 34, 48], and the other is the temporal based methods, which exploit temporal connection between frames such as the temporal order of frames [10, 26, 29] and cross-modal correspondence [23, 39].”
Self-supervised Spatiotemporal Feature Learning by Video Geometric Transformations (2018, methodology): “Most of the 2DConvNet-based methods were trained to learn image representations with temporal supervision such as [19, 23, 32, 35].”
How Useful Is Self-Supervised Pretraining for Visual Tasks? (2020, background): “These methods can take many forms and also rely on a variety of cues such as information across frames in video [46, 38] or across different image modalities [36, 34, 32].”
Charting the Right Manifold: Manifold Mixup for Few-shot Learning (2019, background): “Self-supervision techniques have been successfully applied to diverse set of domains, ranging from robotics to computer vision [31, 12, 57, 55, 45].”
A Survey on Contrastive Self-supervised Learning (2020, unknown): “1 Cross Learn (with Optical Flow) [69] 58.”