Tracking Emerges by Colorizing Videos

By Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, Kevin P. Murphy. 2018

401 citations recorded by the source corpus.

We use large amounts of unlabeled video to learn models for visual tracking without manual human supervision. We leverage the natural temporal coherency of color to create a model that learns to colorize gray-scale videos by copying colors from a reference frame. Quantitative and qualitative experiments suggest that this task causes the model to automatically learn to track visual regions. Although the model is trained without any ground-truth labels, our method learns to track well enough to outperform the latest methods based on optical flow. Moreover, our results suggest that failures to track are correlated with failures to colorize, indicating that advancing video colorization may further improve self-supervised visual tracking.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Representations by Maximizing Mutual Information Across Views (2019, methodology): “While the mutual information bounds in [Vondrick et al., 2018, Eslami et al., 2018] rely on explicit density estimation, our method uses the contrastive bound from CPC [van den Oord et al., 2018], which has been further analyzed by McAllester and Stratos [2018], and Poole et al. [2019].”
Learning Video Representations from Textual Web Supervision (2020, background): “Video (without audio) is already a valuable source of self-supervision, and varied approaches have successfully leveraged supervision from clip and frame ordering [46,17,37,65,67,32], geometry [18,28], motion [52,36], colorization [61], cycle consistency [15,63], and video prediction [43,41,60,59,62”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, background): “Some of these tasks include: temporal ordering of video clips or frames [24, 46, 54, 86], predicting geometric transformations [38], maximizing the mutual information of multiple views [74], predicting motion and appearance [78], predicting the future, the past or a portion of masked input in the fe”
CrossTransformers: spatially-aware few-shot transfer (2020, methodology): “Also particularly relevant are methods that use self-supervision for correspondence [41, 50, 90], which may in future work improve the correspondences that CrossTransformers use.”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “have been used for novel tasks, through pre-training [10, 11], self-supervision [12,13] or additional annotations [14, 15].”
MONet: Unsupervised Scene Decomposition and Representation (2019, neutral): “"Self-supervised" models can also show impressive results on object discovery [Doersch et al., 2015] or tracking [Vondrick et al., 2018]).”
Video Representation Learning by Dense Predictive Coding (2019, background): “Recently [20, 42, 44], leveraged the natural temporal coherency of color in videos, to train a network for tracking and correspondence related tasks.”
Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty (2019, neutral): “In semi-supervised learning, recent self-supervision methods are state-of-the-art [Gidaris et al., 2018, Dosovitskiy et al., 2016, Zhai et al., 2019], and self-supervision is essential in video tasks where annotation is costly [Vondrick et al., 2018, 2016].”
Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction. (2018, methodology): “They also propose to learn spatiotemporal features by colorizing videos with 3DCNN [44].”
Boosting Few-Shot Visual Learning With Self-Supervision (2019, methodology): “Predicting the colors of images [31, 61], the relative position of image patches [6, 40], the random rotation that has been applied to an image [14], or the missing central part of an image [42], are some of the many methods [15, 32, 36, 56, 62] for self-supervised feature learning.”
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, background): “ne learning and has become the key development bottleneck. Recently, to combat costs and eort of annotation, self-supervised arXiv:2003.03186v1 [cs.CV] 6 Mar 2020 2 E. Amrani et al. machine learning [100,49,55,45,92,32,86,85,70] presents new ways to better utilize the abundant unlabeled data on the ”
The Sound of Pixels (2018, background): “Our work builds off efforts to learn perceptual models that are “self-supervised” by leveraging natural contextual signals in images [10,22,33,38,24], videos [46,32,43,44,13,20], and even radio signals [48].”
Spatiotemporal Contrastive Video Representation Learning (2020, background): “Video can help supervise the learning of image representations [68, 49, 65, 24, 53], correspondences [69, 15], and robotic behaviors [56] thanks to its rich content about different views of objects and its motion and tracking cues.”
CoTracker3: Simpler and Better Point Tracking by Pseudo-Labeling Real Videos (2025, neutral): “Accounting for cycle consistency [13, 30, 41] or temporal continuity [9, 43] in videos is another way to obtain a reliable proxy signal to learn correspondences without full supervision, or even to learn generic visual features [10, 40].”
Oops! Predicting Unintentional Action in Video (2019, methodology): “Various types of self-supervised signals have been used to learn strong visual representations, such as spatial arrangement [44], contextual information [14, 68], color [37, 64], the arrow of time [70, 75, 23, 38, 41, 17], future prediction [40, 63, 76, 45], consistency inmotion [3, 26], viewsynthes”
Colorization Transformer (2021, background): “…has important technical uses as a way to learn meaningful representations without explicit supervision (Zhang et al., 2016; Larsson et al., 2016; Vondrick et al., 2018) or as an unsupervised data augmentation technique, whereby diverse semantics-preserving colorizations of labelled images are…”
Online Bag-of-Visual-Words Generation for Unsupervised Representation Learning (2020, neutral): “Numerous pretext tasks have been proposed [3, 10, 16, 28, 47, 50, 51, 73, 75, 80, 84].”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, background): “The learned representation is not only expected to capture the underlying semantic information, but also be transferable to tackle unseen down-stream tasks such as visual recognition, detection, and segmentation [16], visual retrieval [104], and tracking [105].”
Space-Time Correspondence as a Contrastive Random Walk (2020, methodology): “We also consider methods based on the Colorization approach of Vondrick et al. [83], including high-resolution methods: CorrFlow [43] and MAST [42].”
Self-supervised learning of a facial attribute embedding from video (2018, methodology): “…attributes of the source and target frames respectively since the decoder maps from the source frame s to generate the frame s (cid:48) (i.e. it uses pixel RGB values from the source frame s to create the generated s – a similar formulation has been proposed concurrently to this work by [54]).”