Temporal Cycle-Consistency Learning

By Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, Andrew Zisserman. 2019

324 citations recorded by the source corpus.

We introduce a self-supervised representation learning method based on the task of temporal alignment between videos. The method trains a network using temporal cycle-consistency (TCC), a differentiable cycle-consistency loss that can be used to find correspondences across time in multiple videos. The resulting per-frame embeddings can be used to align videos by simply matching frames using nearest-neighbors in the learned embedding space. To evaluate the power of the embeddings, we densely label the Pouring and Penn Action video datasets for action phases. We show that (i) the learned embeddings enable few-shot classification of these action phases, significantly reducing the supervised training requirements; and (ii) TCC is complementary to other methods of self-supervised learning in videos, such as Shuffle and Learn and Time-Contrastive Networks. The embeddings are also used for a number of applications based on alignment (dense temporal correspondence) between video pairs, including transfer of metadata of synchronized modalities between videos (sounds, temporal semantic labels), synchronized playback of multiple videos, and anomaly detection. Project webpage: https://sites.google.com/view/temporal-cycle-consistency .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, methodology): “For example, prior approaches have successfully lever-aged supervision from clip and frame ordering [47, 16, 38, 66, 68, 33], geometry [17, 29], motion [53, 37], colorization [62], cycle consistency [14, 64], video prediction [44, 42, 61, 60, 63], tempo [69], and in-video consistency [22, 54].”
Audio-Visual Instance Discrimination with Cross-Modal Agreement (2020, background): “They can use the spatial structure in images [15, 21, 53, 86], color [13, 40, 41, 87], temporal information in videos [17, 19, 25, 31, 44, 50, 54, 60, 82] among other sources”
Spatiotemporal Contrastive Video Representation Learning (2020, background): “Video can help supervise the learning of image representations [42, 31, 40, 15, 33], correspondences [43, 8], and robotic behaviors [35] thanks to its rich content about different views of objects and its motion and tracking cues.”
Whitening for Self-Supervised Representation Learning (2020, methodology): “Misra et al. (2016) extended this idea using the relative temporal order of 3 frames, while Dwibedi et al. (2019) used a temporal cycle consistency for self-supervision, which is based on comparing two videos sharing the same semantics and computing inter-video frame-to-frame nearest neighbour…”
Efficient Training of Visual Transformers with Small Datasets (2021, methodology): “001, as suggested in [5], works well in our scenario, thus we adopted it for all the experiments involving L drloc.”
Learning Video Representations using Contrastive Bidirectional Transformer (2019, background): “temporal cycle consistency (Wang et al., 2019b; Dwibedi et al., 2019) or colorization (Vondrick et al.”
Video Representation Learning with Visual Tempo Consistency (2020, background): “…been explored for self-supervised video representation learning, such as modeling the cycle-consistency between two videos of the same category [10], modeling the cycle-consistency of time [45], predicting the temporal order of frames [12, 29, 33, 46], predicting future motion dynamics and…”
Primal Wasserstein Imitation Learning (2020, methodology): “This distance is learned offline through selfsupervised learning: we construct an embedding of the frames using Temporal Cycle-Consistency Learning (TCC) (Dwibedi et al., 2019); TCC builds an embedding by aligning frames of videos of a task coming from multiple examples (in this case the demonstrati”
XIRL: Cross-embodiment Inverse Reinforcement Learning (2021, methodology): “In this work, we propose to enable agents to imitate video demonstrations of experts – including ones with different embodiments – by using a task-specific, embodiment-invariant reward formulation trained via temporal cycle-consistency (TCC) [7].”
Space-Time Correspondence as a Contrastive Random Walk (2020, methodology): “Self-supervised Correspondence and Cycle-consistency in Time Our approach is directly inspired by recent work that uses cycle-consistency [99, 19] in time as supervisory signal for learning visual representations from video [88, 84].”
COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning (2020, background): “TCC [38] employs cycle-consistency for temporal video alignment.”
Few-Shot Segmentation via Cycle-Consistent Transformer (2021, background): “Our work is partially inspired by cycle-consistency learning [43, 7] that is explored in various computer vision areas.”
Oops! Predicting Unintentional Action in Video (2019, background): “…such as spatial arrangement [42], contextual information [12, 66], color [35, 62], the arrow of time [68, 73, 21, 36, 39, 15], future prediction [38, 61, 74, 43], consistencyinmotion[3,24], viewsynthesis[81,80], spatio-temporal coherence [58, 67, 14, 37, 33], and predictive coding [43, 55].”
Temporal Attentive Alignment for Large-Scale Video Domain Adaptation (2019, background): “Some papers related to cycleconsistency [49, 8] introduce self-supervised methods for learning visual correspondence between images or videos from unlabeled videos.”
Counting Out Time: Class Agnostic Video Repetition Counting in the Wild (2020, methodology): “Self-supervised learning methods for video embeddings, e.g. Shufﬂe and Learn [31], Odd-One-Out networks [14], DPC [17], TCC [11] and TCN [39] are not designed to handle repetitions in sequences.”
Learning Temporal Dynamics from Cycles in Narrated Video (2021, unknown): “Dwibedi et al .”
MimicPlay: Long-Horizon Imitation Learning by Watching Human Play (2023, background): “Potential future directions include temporal contrastive learning [54] and cycle consistency learning [55] from videos.”
Listen to Look: Action Recognition by Previewing Audio (2019, unknown): “Motivated by the locally-smooth action feature space [12] and based on our empirical observation that neighboring video features can be linearly approximated well, we synthesize the missing image and audio features by computationally cheap linear interpolation to generate the full feature sequences ”