UvA-DARE (Digital Academic Repository) Self-Supervised Video Representation Learning With Odd-One-Out Networks

By Basura Fernando, Hakan Bilen, Efstratios Gavves, Stephen Gould

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “For example, [40, 35, 16, 5, 64] make use of the order in which frames appear in a video.”
Learning Visual Representations with Caption Annotations (2020, background): “At the other extreme of visual representation learning, self-supervised learning methods [7,15,16,18,66,67] do not require annotations and fabricate proxy labels from the data itself.”
Learning Video Representations from Textual Web Supervision (2020, methodology): “For example, prior approaches have successfully lever-aged supervision from clip and frame ordering [47, 16, 38, 66, 68, 33], geometry [17, 29], motion [53, 37], colorization [62], cycle consistency [14, 64], video prediction [44, 42, 61, 60, 63], tempo [69], and in-video consistency [22, 54].”
Self-Supervised MultiModal Versatile Networks (2020, background): “In videos, works have also leveraged the temporal dimension [23, 46, 56, 91].”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, neutral): “Some of these tasks include: temporal ordering of video clips or frames [24, 46, 54, 86], predicting geometric transformations [38], maximizing the mutual information of multiple views [74], predicting motion and appearance [78], predicting the future, the past or a portion of masked input in the fe”
Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “For video data, these pretext tasks learn representations by ordering video frames [1, 18, 34, 41, 51, 79, 83], tracking [62, 77], or using cross-modal signals like audio [2, 3, 19, 36, 60, 61].”
Multi-task Self-Supervised Visual Learning (2017, methodology): “[10]); and we can experiment with methods for dynamically weighting the importance of tasks in the optimization.”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, neutral): “One can also use the temporal structure in a video for self-supervised methods [21, 27, 41, 46, 47].”
Time-Contrastive Networks: Self-Supervised Learning from Video (2017, background): “A number of prior works use temporal coherence [28, 29, 30, 31].”
Objects that Sound (2017, neutral): “As the labels are constructed directly from the data itself, this is an example of “self-supervision” [1, 10, 11, 12, 14, 20, 21, 25, 32, 35],”
The Challenges of Continuous Self-Supervised Learning (2022, methodology): “coding [36, 54, 55], inpainting [59] and colorization [18, 34, 86], to methods that learn representations predictive of spatial context [19, 22, 53], temporal context [21, 47, 58, 61, 79, 80], or concurrent modalities like audio [4, 52, 57, 70], text [17, 23, 62] or speech [43, 44].”
Unsupervised Representation Learning by Sorting Sequences (2017, methodology): “Fernando et al. [8] exploit a similar notion of order veriﬁcation to learn video representation.”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, background): “Examples include predicting the future frame [86], motion and appearance statistics [95], speed [9, 96] and encodings [62, 42, 43], sorting frames or video clips [58, 103, 49, 35].”
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (2018, background): “predicting the temporal ordering in a video [17,18,19].”
Video Representation Learning by Dense Predictive Coding (2019, background): “Other than the predictive tasks reviewed in the introduction, another class of proxy tasks is based on temporal sequence ordering of the frames [6, 26, 45].”
Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization (2018, methodology): “[42] used temporal context for self-supervised pre-training, while Misra et al.”
Learning and Using the Arrow of Time (2018, methodology): “Others have used the temporal ordering of frames as a supervisory signal for learning an embedding [15], for self-supervision training of a ConvNet [6, 13], and for construction of a representation for action recognition [7].”
Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction. (2018, methodology): “Fernando et al. propose to learn video representations by odd-one-out networks to identify the odd element from a set of related elements with a 2DCNN [10].”
Cross and Learn: Cross-Modal Self-Supervision (2018, methodology): “elds inferior results. We conduct our evaluation with CaﬀeNet on split 1 of UCF-101 and HMDB-51 only using UCF-101 for pretraining. Using Stack of Frame Diﬀerences as Second Modality. In the works of [9] and [21] reliable results were achieved using stack of frame diﬀerences (SOD) for action recogni”