End-to-End Learning of Visual Representations From Uncurated Instructional Videos

By Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, Andrew Zisserman. 2019

799 citations recorded by the source corpus.

Annotating videos is cumbersome, expensive and not scalable. Yet, many strong video models still rely on manually annotated data. With the recent introduction of the HowTo100M dataset, narrated videos now offer the possibility of learning video representations without manual supervision. In this work we propose a new learning approach, MIL-NCE, capable of addressing mis- alignments inherent in narrated videos. With this approach we are able to learn strong video representations from scratch, without the need for any manual annotation. We evaluate our representations on a wide range of four downstream tasks over eight datasets: action recognition (HMDB-51, UCF-101, Kinetics-700), text-to- video retrieval (YouCook2, MSR-VTT), action localization (YouTube-8M Segments, CrossTask) and action segmentation (COIN). Our method outperforms all published self-supervised approaches for these tasks as well as several fully supervised baselines.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “Several works have explored using dense spoken natural language supervision for videos (Miech et al., 2019; 2020b).”
Learning Video Representations from Textual Web Supervision (2020, methodology): “Prior works have leveraged ambient sound [52, 5, 4, 71, 34, 51, 55], dialogue [49], and narration [70, 2, 72, 73, 46, 45, 3], all of which of which serve as useful signals.”
Self-Supervised MultiModal Versatile Networks (2020, methodology): “The scores are tied on Vision-Text tasks, with the 3-modality network winning on MSRVTT but losing on YC2.”
Contrastive Multiview Coding (2019, background): “The idea of contrastive learning has also started to spread over many other tasks in various other domains [74, 86, 61, 76, 48, 38, 75].”
Self-supervised Pretraining of Visual Features in the Wild (2021, background): “Some works have explored unsupervised training in the wild for images [6, 13, 19] and videos [36].”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “Other methods have also attempted zero-shot on MSR-VTT text-to-video retrieval (Xu et al. 2021; Miech et al. 2020; Bain et al. 2021), but these have all been outperformed by Portillo-Quintero et al. (2021).”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “[74] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman.”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, background): “To enhance the performance, there have been parallel interests in bringing in more modalities from raw video inputs [31,47,48] and end-to-end training till the raw pixels [18,19,25,49], both aiming to elevate video representations for VidL modeling.”
Florence: A New Foundation Model for Computer Vision (2021, methodology): “(∗: Feature extracted from the pre-trained model (Miech et al., 2020), followed by another stage of video-and-language pre-training) The pretraining data used in these existing methods include HowTo100M (Miech et al.”
MERLOT: Multimodal Neural Script Knowledge Models (2021, background): “Of late, works have sought to learn multimodal representations transferable to many tasks from uncurated sets of (usually how-to) videos [68, 93, 94, 69, 115, 8, 6, 3]; generally these are applied to video understanding tasks like activity recognition.”
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (2021, methodology): “We use S3D (Xie et al., 2018; Miech et al., 2020), which is pre-trained via selfsupervised learning on the Howto100M dataset.”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, methodology): “using loss functions such as MIL-NCE [35] or using the raw audio [1, 48] directly to increase robustness.”
Unifying Vision-and-Language Tasks via Text Generation (2021, background): “…et al., 2020; Radford et al., 2021; Zhang et al., 2021) and video+text pretraining models (Sun et al., 2019b;a; Li et al., 2020a; Zhu & Yang, 2020; Miech et al., 2020) have also shown to perform better than previous non-pretraining approaches (Yu et al., 2018a; Anderson et al., 2018; Kim et al.,…”
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers (2021, background): “(5)) require sampling many negative examples to achieve good performance and thus can be computationally expensive ( e.g. , Tian et al., 2019; Miech et al., 2020 This limits the number of negatives available for these models to the maximum batch size that ﬁts in memory.”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, positive): “We use a frozen S3D [82] pretrained on HowTo100M [58] using MIL-NCE [57].”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, neutral): “, 2019a), MIL-NCE (Miech et al., 2020), ActBERT (Zhu and Yang, 2020) and UniViLM (Luo et al.”
UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, methodology): “retrieval orienting operation, which is to align the space of representation between text and video. Considering the misalignment between the text and video clip in narrated videos, we adopt MIL-NCE (Miech et al., 2020) on T and V as our joint objective, L Joint() = E (t;v)˘B logMIL-NCE(t;v); (5) MI”