Unsupervised Learning using Sequential Verification for Action Recognition

By Ishan Misra, C. Lawrence Zitnick, Martial Hebert. 2016

68 citations recorded by the source corpus.

In this paper, we consider the problem of learning a visual representation from the raw spatiotemporal signals in videos for use in action recognition. Our representation is learned without supervision from semantic labels. We formulate it as an unsupervised sequential verification task, i.e., we determine whether a sequence of frames from a video is in the correct temporal order. With this simple task and no semantic labels, we learn a powerful unsupervised representation using a Convolutional Neural Network (CNN). The representation contains complementary information to that learned from supervised image datasets like ImageNet. Qualitative results show that our method captures information that is temporally varying, such as human pose. When used as pre-training for action recognition, our method gives significant gains over learning without external data on benchmark datasets like UCF101 and HMDB51. Our method can also be combined with supervised representations to provide an additional boost in accuracy for action recognition. Finally, to quantify its sensitivity to human pose, we show results for human pose estimation on the FLIC dataset that are competitive with approaches using significantly more supervised training data.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Human-Computer Interaction

Citations in the corpus

Generating Videos with Scene Dynamics (2016, methodology): “Moreover, our model slightly outperforms another unsupervised video representation [45] despite using an order of magnitude fewer learned parameters and only 64 × 64 videos.”
Colorization as a Proxy Task for Visual Understanding (2017, methodology): “Another method that uses a triplet loss presents three frames and tries to predict if they are correctly ordered [25].”
UvA-DARE (Digital Academic Repository) Self-Supervised Video Representation Learning With Odd-One-Out Networks (critical): “our O3N learning-based features obtains score almost 10% higher in UCF101 than the second best method reported in the literature [31] that relies on sequential verification.”
Cross and Learn: Cross-Modal Self-Supervision (2018, background): “Approaches which use video data mainly exploit temporal context as source of supervision [40,24,4,5,2].”
Improving Spatiotemporal Self-Supervision by Deep Reinforcement Learning (2018, methodology): “In previous works [31, 27, 16, 33, 5], for each training sample one permutation is randomly selected from a large set of candidate permutations ψi ∈ Ψ .”
Going deeper into action recognition: A survey (2016, background): “In Misra et al. (2016), how temporal coherency can be used to train deep models for action recognition and pose estimation has been explored.”
Time-Contrastive Networks: Self-Supervised Learning from Multi-view Observation (2017, methodology): “1), the single-view TC loss or the Shuffle & Learn loss[14].”
SfM-Net: Learning of Structure and Motion from Video (2017, methodology): “We use differentiable image warping proposed in the spatial trans-former work [16] and compute color constancy loss in a fully differentiable manner.”
Self-Supervised Learning in Remote Sensing: A review (2022, background): “Frame order prediction in SSL can be further divided into two distinct approaches: 1) temporal order verification [144]–[147] and 2) temporal order recognition [66], [148], [149].”