Improving Spatiotemporal Self-Supervision by Deep Reinforcement Learning

By Uta Büchler, Biagio Brattoli, Björn Ommer. 2018

121 citations recorded by the source corpus.

Self-supervised learning of convolutional neural networks can harness large amounts of cheap unlabeled data to train powerful feature representations. As surrogate task, we jointly address ordering of visual data in the spatial and temporal domain. The permutations of training samples, which are at the core of self-supervision by ordering, have so far been sampled randomly from a fixed preselected set. Based on deep reinforcement learning we propose a sampling policy that adapts to the state of the network, which is being trained. Therefore, new permutations are sampled according to their expected utility for updating the convolutional feature representation. Experimental evaluation on unsupervised and transfer learning tasks demonstrates competitive performance on standard benchmarks for image and video classification and nearest neighbor retrieval.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “For example, [40, 35, 16, 5, 64] make use of the order in which frames appear in a video.”
Contrastive Multiview Coding (2019, neutral): “Added references to Time Contrastive Networks [53] and Local Aggregation [64].”
Cross and Learn: Cross-Modal Self-Supervision (2018, background): “Approaches which use video data mainly exploit temporal context as source of supervision [40,24,4,5,2].”
Audio-Visual Instance Discrimination with Cross-Modal Agreement (2020, neutral): “8 ST Order [9] 1×227 58.”
Human Action Recognition and Prediction: A Survey (2018, methodology): “On the only RGB input setting, compared with CNN-based approaches, Our C3D outperforms deep networks [18] and spatial stream network in [36] by 19.”
Self-Supervised Spatiotemporal Learning via Video Clip Order Prediction (2019, methodology): “Previous works [27, 1] select several particular orders from all possible orders either determinately or adaptively.”
Self-supervised Spatiotemporal Feature Learning by Video Geometric Transformations (2018, background): “Learning from videos and achieved very good performance on the two datasets [2].”
Self-supervised Co-training for Video Representation Learning (2020, unknown): “the same semantic categories. 8 Method Date Dataset UCF HMDB R@1 R@5 R@10 R@20 R@1 R@5 R@10 R@20 Jigsaw [47] 2016 UCF 19.7 28.5 33.5 40.0 - - - - OPN [42] 2017 UCF 19.9 28.7 34.0 40.6 - - - - Buchler [8] 2018 UCF 25.7 36.2 42.2 49.2 - - - - VCOP [66] 2019 UCF 14.1 30.3 40.4 51.1 7.6 22.9 34.4 48.8 V”
Self-supervised Video Representation Learning by Pace Prediction (2020, methodology): “It can be seen from the table that with the increase of the maximum pace, namely the number of training classes, the accuracy of the downstream action recognition task keeps increase, until p = [1, 4].”
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey (2019, methodology): “1 RL [35] Context 58.”