Video Jigsaw: Unsupervised Learning of Spatiotemporal Context for Video Action Recognition

By Unaiza Ahsan, Rishi Madhok, Irfan Essa. 2018

121 citations recorded by the source corpus.

We propose a self-supervised learning method to jointly reason about spatial and temporal context for video recognition. Recent self-supervised approaches have used spatial context [9, 34] as well as temporal coherency [32] but a combination of the two requires extensive preprocessing such as tracking objects through millions of video frames [59] or computing optical flow to determine frame regions with high motion [30]. We propose to combine spatial and temporal context in one self-supervised framework without any heavy preprocessing. We divide multiple video frames into grids of patches and train a network to solve jigsaw puzzles on these patches from multiple frames. So the network is trained to correctly identify the position of a patch within a video frame as well as the position of a patch over time. We also propose a novel permutation strategy that outperforms random permutations while significantly reducing computational and memory constraints. We use our trained network for transfer learning tasks such as video activity recognition and demonstrate the strength of our approach on two benchmark video action recognition datasets without using a single frame from these datasets for unsupervised pretraining of our proposed video jigsaw network.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “For video data, these pretext tasks learn representations by ordering video frames [1, 18, 34, 41, 51, 79, 83], tracking [62, 77], or using cross-modal signals like audio [2, 3, 19, 36, 60, 61].”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “2) Semi-Supervised Learning: SSL and semi-supervised learning are contrasting paradigms that can be effectively combined.”
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey (2019, methodology): “Mehtod SubCategory Code Contribution VideoGAN [85] Generation 3 Forerunner of video generation with GAN MocoGAN [86] Generation 3 Decomposing motion and content for video generation with GAN TemporalGAN [144] Generation 3 Decomposing temporal and image generator for video generation Video Colorizati”
Self-supervised Spatiotemporal Representation Learning by Exploiting Video Continuity (2021, background): “(Wang et al. 2019), identifying the rotation angle of video clips (Jing et al. 2018) or solving spatiotemporal jigsaw puzzles (Ahsan, Madhok, and Essa 2019; Kim, Cho, and Kweon 2019), etc.”
Modeling Long-horizon Tasks as Sequential Interaction Landscapes (2020, methodology): “[1] propose a self-supervised method to identify the spatio-temporal context in videos.”
GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning (2021, methodology): “…most of these methods construct pseudo labels automatically and train on auxiliary tasks, including image jigsaw (Noroozi and Favaro, 2016; Ahsan et al., 2019), inpainting (Pathak et al., 2016), super resolution (Ledig et al., 2017), etc. Inspired by these works, we design two…”