Unsupervised Representation Learning by Sorting Sequences

By Hsin-Ying Lee, Jia-Bin Huang, Maneesh Kumar Singh, Ming-Hsuan Yang. 2017

570 citations recorded by the source corpus.

We present an unsupervised representation learning approach using videos without semantic labels. We leverage the temporal coherence as a supervisory signal by formulating representation learning as a sequence sorting task. We take temporally shuffled frames (i.e., in non-chronological order) as inputs and train a convolutional neural network to sort the shuffled sequences. Similar to comparison-based sorting algorithms, we propose to extract features from all frame pairs and aggregate them to predict the correct order. As sorting shuffled image sequence requires an understanding of the statistical temporal structure of images, training with such a proxy task allows us to learn rich and generalizable visual representation. We validate the effectiveness of the learned representation using our method as pre-training on high-level recognition problems. The experimental results show that our method compares favorably against state-of-the-art methods on action recognition, image classification, and object detection tasks.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “For example, [40, 35, 16, 5, 64] make use of the order in which frames appear in a video.”
Learning Video Representations from Textual Web Supervision (2020, methodology): “For example, prior approaches have successfully lever-aged supervision from clip and frame ordering [47, 16, 38, 66, 68, 33], geometry [17, 29], motion [53, 37], colorization [62], cycle consistency [14, 64], video prediction [44, 42, 61, 60, 63], tempo [69], and in-video consistency [22, 54].”
Self-Supervised MultiModal Versatile Networks (2020, background): “In videos, works have also leveraged the temporal dimension [23, 46, 56, 91].”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, background): “ (sectionA). Finally, we also compare to the single-instance learning baseline where we concatenate all of the candidate narrations as one longer narration Method Dataset MM Model Frozen HMDB UCF OPN [46] UCF 7 VGG 7 23.8 59.6 Shufﬂe & Learn [54]* K600 7 S3D 7 35.8 68.7 Wang et al. [78] K400 Flo”
Contrastive Multiview Coding (2019, neutral): “3 OPN [44] 1 56.”
Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “For video data, these pretext tasks learn representations by ordering video frames [1, 18, 34, 41, 51, 79, 83], tracking [62, 77], or using cross-modal signals like audio [2, 3, 19, 36, 60, 61].”
A critical analysis of self-supervision, or what we can learn from a single image (2019, background): “Typical pretext tasks are based on temporalcontext (Misra et al., 2016; Wei et al., 2018; Lee et al., 2017; Sermanet et al., 2018), spatio-temporal 1Example applications that only rely on low-level feature extractors include template matching (Kat et al., 2018; Talmi et al., 2017) and style…”
Self-Supervised GANs via Auxiliary Rotation Loss (2018, methodology): “It has been widely used in the video domain [39, 40], the robotics domain [41, 42] and the image domain [24, 35].”
Invariant Information Clustering for Unsupervised Image Classification and Segmentation (2018, background): “The latter includes objectives such as generative autoencoder image reconstruction [46], triplets [44] and spatial-temporal order or context prediction [36, 12, 16], for example predicting patch proximity [29], solving jig-saw puzzles [40] and inpainting [41].”
Self-labelling via simultaneous clustering and representation learning (2019, background): “Typical pretext tasks are based on temporal-context (Misra et al., 2016; Wei et al., 2018; Lee et al., 2017; Sermanet et al., 2018), spatio-temporal cues (Isola et al., 2015; Gao et al., 2016; Wang et al., 2017), foreground-background separation via video segmentation (Pathak et al., 2017),…”
Transformers in Vision: A Survey (2021, methodology): “The context-based pretext methods solve problems such as a jigsaw puzzle on image patches [5, 101, 150], masked object classiication [182], predict geometric transformation such as rotation [67, 94], or verify temporal sequence of video frames [114, 144, 218].”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, background): “Examples include predicting the future frame [86], motion and appearance statistics [95], speed [9, 96] and encodings [62, 42, 43], sorting frames or video clips [58, 103, 49, 35].”
High-Fidelity Image Generation With Fewer Labels (2019, positive): “This approach was successfully applied to different data modalities, such as images (Doersch et al., 2015; Caron et al., 2018), video (Agrawal et al., 2015; Lee et al., 2017), and robotics (Jang et al., 2018; Pinto & Gupta, 2016).”
Improvements to Context Based Self-Supervised Learning (2017, methodology): “For instance, [25] shuffled four consecutive video frames to create 12 classes for prediction.”
Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction. (2018, methodology): “take shuffled video frames as input to a 2DCNN to sort the sequence [26].”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, methodology): “A variety of tasks, focused primarily on text-video modelling, have been applied to these datasets including: task segmentation [19–21], reference resolution [22, 23], action segmentation [5], video clip ordering [24–27], and action recognition [28–30].”
Cross and Learn: Cross-Modal Self-Supervision (2018, methodology): “Similarly to [21,24] the latter sampling is weighted by the average flow magnitudes in a pair which reduces the number of training samples without meaningful information.”
Boosting Few-Shot Visual Learning With Self-Supervision (2019, methodology): “Predicting the colors of images [31, 61], the relative position of image patches [6, 40], the random rotation that has been applied to an image [14], or the missing central part of an image [42], are some of the many methods [15, 32, 36, 56, 62] for self-supervised feature learning.”