HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips

By Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, Josef Sivic. 2019

1,493 citations recorded by the source corpus.

Learning text-video embeddings usually requires a dataset of video clips with manually provided captions. However, such datasets are expensive and time consuming to create and therefore difficult to obtain on a large scale. In this work, we propose instead to learn such embeddings from video data with readily available natural language annotations in the form of automatically transcribed narrations. The contributions of this work are three-fold. First, we introduce HowTo100M: a large-scale dataset of 136 million video clips sourced from 1.22M narrated instructional web videos depicting humans performing and describing over 23k different visual tasks. Our data collection procedure is fast, scalable and does not require any additional manual annotation. Second, we demonstrate that a text-video embedding trained on this data leads to state-of-the-art results for text-to-video retrieval and action localization on instructional video datasets such as YouCook2 or CrossTask. Finally, we show that this embedding transfers well to other domains: fine-tuning on generic Youtube videos (MSR-VTT dataset) and movies (LSMDC dataset) outperforms models trained on these datasets alone. Our dataset, code and models are publicly available.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Several works have explored using dense spoken natural language supervision for videos (Miech et al., 2019; 2020b).”
RareAct: A video dataset of unusual interactions (2020, methodology): “To assess the compositionality aspect of action recognition models, we consider combinations of verbs and nouns that satisfy the following two criteria: (i) The verbs and nouns never or rarely co-occur together in the HowTo100M [8] textual corpus; and (ii) The verbs and nouns frequently appear separ”
Learning Video Representations from Textual Web Supervision (2020, methodology): “Using this process, we collect a dataset of 70 million videos, which is comparable in scale to the recent HowTo100M dataset [46], but includes paired metadata, is drawn from a higher number of unique videos (70M vs 1M), and is not limited to only instructional videos (Section 3).”
Self-Supervised MultiModal Versatile Networks (2020, methodology): “2) is trained on 45981 frames of the HowTo100M [54] training set, where the static videos (ingested by the original video network to produce the regression targets for the deflated image network) are 32-frame long to match the video length used during self-supervised training; the Adam optimizer [39”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, positive): “How to learn despite noisy supervision? To address the aforementioned issues, we propose to consider multiple options for matching a video and a narration instead of only comparing a single video x with a single narration y as done in [52].”
All in One: Exploring Unified Video-Language Pre-Training (2022, methodology): “5M [4], HowTo100M [29] and YTTemporal 180M [48] for Pre-training.”
Multiview Transformers for Video Recognition (2022, methodology): “Moreover, our model with ImageNet-21K pretraining even outperforms VATT [2], which was pretrained on HowTo100M [47], a dataset consisting of around 100M video clips.”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “Weplan to release only video IDs for download, following prior work [1, 83].”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “Previous works [5, 6, 14–17] attempt cross-modal fusion over dense video features and text features to tackle VidL tasks, but suffer from domain disconnection due to offline feature extraction [16, 18].”
Florence: A New Foundation Model for Computer Vision (2021, methodology): “, 2020), followed by another stage of video-and-language pre-training) The pretraining data used in these existing methods include HowTo100M (Miech et al., 2019), AudioSet (Gemmeke et al.”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Second, it means that the HowTo100M videos do contain some instructional videos from less-popular channels.”
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (2021, methodology): “Average Recall Joint Alignment Alayrac (Alayrac et al., 2016) 13.3 Zhukov (Zhukov et al., 2019) 22.4 Supervised (Zhukov et al., 2019) 31.6 HowTo100M (Miech et al., 2019) 33.6 MIL-NCE (Miech et al., 2020) 40.5 UniVL (Luo et al., 2020) 42.0 Pairwise Matching ActBERT (Zhu and Yang, 2020) 41.4 VLM (task”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, positive): “In this work, we exploit noisy correlations between videos and narrations in unlabeled instructional videos from the recent HowTo100M dataset [58].”
ActBERT: Learning Global-Local Video-Text Representations (2020, methodology): “Following [26], we evaluate the text-video clip retrieval task on the validation clips of YouCook2.”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “To empower the model with richer knowledge such as contextual understanding of dynamic social interactions between multi-characters and dramatic scene/event evolvement, we jointly train HERO on two diverse datasets: HowTo100M dataset (containing 22k narrated instructional videos) (Miech et al., 2019”
UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, methodology): “We ﬁrst pre-train our model on the large scale dataset HowTo100M (Miech et al., 2019), then ﬁne-tune our pre-trained model on two downstream multimodal tasks including text-based video retrieval and multimodel video captioning.”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Existing collections like Kinetics [108], AVA [92], UCF [207], Ac-tivityNet [61], HowTo100M [157], ImageNet [49], and COCO [143] focus on third-person Web data, which have the benefit and bias of a human photographer.”