Learning Video Representations from Textual Web Supervision

By Jonathan C. Stroud, David A. Ross, Chen Sun, Jia Deng, Rahul Sukthankar, Cordelia Schmid. 2020

51 citations recorded by the source corpus.

Videos found on the Internet are paired with pieces of text, such as titles and descriptions. This text typically describes the most important content in the video, such as the objects in the scene and the actions being performed. Based on this observation, we propose to use such text as a method for learning video representations. To accomplish this, we propose a data collection process and use it to collect 70M video clips shared publicly on the Internet, and we then train a model to pair each video with its associated text. We fine-tune the model on several down-stream action recognition tasks, including Kinetics, HMDB-51, and UCF-101. We find that this approach is an effective method of pretraining video representations. Specifically, it leads to improvements over from-scratch training on all benchmarks, outperforms many methods for self-supervised and webly-supervised video representation learning, and achieves an improvement of 2.2% accuracy on HMDB-51.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Stroud et al. (2020) explores large scale representation learning by training a system to pair descriptive text with videos instead of images.”
VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “[76] on learning video representations using paired textual metadata, however they solely operate and evaluate their method on video tasks.”
Multiview Transformers for Video Recognition (2022, methodology): “And when we utilize the Weak Textual Supervision (WTS) dataset of [61]”
CLIP2Video: Mastering Video-Text Retrieval via Image CLIP (2021, background): “Others [18, 14, 35, 19] collect videos and accompanied text information from YouTube and Instagram to to learn spatio-temporal features in an efﬁcient weakly-supervised manner.”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, methodology): “Similarly, other methodologies [62], [218] are also founded on joint video and audio modalities while certain studies [219], [220], [221] incorporated both video and text modalities.”
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (2023, unknown): “0 52K 360P WTS70M [49] Metadata action 70M 70M 10 - 194K HowTo100M [2] ASR instruct 1.”
DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization (2021, background): “…models (Sun et al., 2019; Miech et al., 2019; Zhu and Yang, 2020; Miech et al., 2020; Li et al., 2020b; Luo et al., 2020; Huang et al., 2020; Stroud et al., 2020) have also shown promising results on many vision and language tasks (Antol et al., 2015; Xu et al., 2016; Zhou et al., 2017).”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, background): “, from web data [307] or narrated materials [308], [309], [310], [311], [312], [313]) to guide unsupervised representation learning by aligning images and languages in the shared latent space, as exemplified by CLIP [312] and ALIGN [313].”