Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification

By Ishan Misra, C. Lawrence Zitnick, Martial Hebert. 2016

902 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Human-Computer Interaction

Citations in the corpus

Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “Nevertheless, discriminative tasks useful for learning representations for action recognition, such as predicting whether a sequence of frames is played forward or backward [60], verifying whether the frames are ordered or shuffled [37], or predicting features corresponding to future frames [20], ca”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, neutral): “, 2018; Arandjelovic & Zisserman, 2018; 2017), video (Jing & Tian, 2018; Misra et al., 2016), and robotic manipulation (Pinto & Gupta, 2016; Pinto et al.”
VideoBERT: A Joint Model for Video and Language Representation Learning (2019, background): “, [18]), and try to predict one from the other (see e.”
Learning Video Representations from Textual Web Supervision (2020, background): “Video (without audio) is already a valuable source of self-supervision, and varied approaches have successfully leveraged supervision from clip and frame ordering [46,17,37,65,67,32], geometry [18,28], motion [52,36], colorization [61], cycle consistency [15,63], and video prediction [43,41,60,59,62”
Self-Supervised MultiModal Versatile Networks (2020, background): “In videos, works have also leveraged the temporal dimension [23, 46, 56, 91].”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, neutral): “Shuffle & Learn [54]* K600 S3D 35.”
Contrastive Multiview Coding (2019, neutral): “9 Shuffle and Learn [50] 1 50.”
Learning Features by Watching Objects Move (2016, background): “ing the input [4, 20, 44], predicting the pixels of the next frame in a video stream [17], metric learning on object track endpoints [46], temporally ordering shuffled frames from a video [29], and spatially ordering patches from a static image [8, 30].”
Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “For video data, these pretext tasks learn representations by ordering video frames [1, 18, 34, 41, 51, 79, 83], tracking [62, 77], or using cross-modal signals like audio [2, 3, 19, 36, 60, 61].”
A critical analysis of self-supervision, or what we can learn from a single image (2019, background): “Typical pretext tasks are based on temporalcontext (Misra et al., 2016; Wei et al., 2018; Lee et al., 2017; Sermanet et al., 2018), spatio-temporal”
Multi-task Self-Supervised Visual Learning (2017, background): “One way to simplify the problem is to ask a network to temporally order a set of frames sampled from a video [23].”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, neutral): “One can also use the temporal structure in a video for self-supervised methods [21, 27, 41, 46, 47].”
Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction (2016, methodology): “[29] – – – [29] 42.”
Multi-Task Self-Supervised Learning for Robust Speech Recognition (2020, neutral): “A related field that is gaining popularity is self-supervised learning, where targets are computed from the signal itself [5, 6] by applying known transformations to the input data.”
MERLOT: Multimodal Neural Script Knowledge Models (2021, neutral): “In addition to extrapolation, past work has studied deshuffling objectives in video frames [70, 102], though this has mostly been focused on the visual modality.”
ActBERT: Learning Global-Local Video-Text Representations (2020, background): “While supervised learning has been successful in a variety of computer vision tasks [17, 9, 38, 29], self-supervised representation learning from unlabeled data has attracted increasing attention in recent years [4, 27].”
Contrastive Bidirectional Transformer for Temporal Representation Learning (2019, background): “Most of the recent efforts on learning deep representations for videos have been focusing on capturing the appearance and local motion pattern from short video clips [22, 13, 11], in an supervised [26, 5, 35] or self-supervised [15, 31] fashion.”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, neutral): “Many self-supervised methods manipulate the input data to extract a supervised signal in the form of a pretext task [1, 13, 26, 28, 30, 36, 38, 41, 42, 47, 48, 57].”