Moments in Time Dataset: One Million Videos for Event Understanding

By Mathew Monfort, Bolei Zhou, Sarah Adel Bargal, Alex Andonian, Tom Yan, Kandan Ramakrishnan, Lisa Gottesfeld Brown, Quanfu Fan, D. Gutfreund, Carl Vondrick and 1 more. 2018

632 citations recorded by the source corpus.

We present the Moments in Time Dataset, a large-scale human-annotated collection of one million short videos corresponding to dynamic events unfolding within three seconds. Modeling the spatial-audio-temporal dynamics even for actions occurring in 3 second videos poses many challenges: meaningful events do not include only people, but also objects, animals, and natural phenomena; visual and auditory events can be symmetrical in time (“opening” is “closing” in reverse), and either transient or sustained. We describe the annotation process of our dataset (each video is tagged with one action or activity label among 339 different classes), analyze its scale and diversity in comparison to other large-scale video datasets for action recognition, and report results of several baseline models addressing separately, and jointly, three modalities: spatial, temporal and auditory. The Moments in Time dataset, designed to have a large coverage and diversity of events in both visual and auditory modalities, can serve as a new challenge to develop models that scale to the level of complexity and abstract reasoning that a human processes on a daily basis.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

In Signal Processing

Citations in the corpus

VideoBERT: A Joint Model for Video and Language Representation Learning (2019, background): “, [9, 19, 36, 7]) to train convolutional neural networks for video classification.”
Multiview Transformers for Video Recognition (2022, background): “Inspired by ViT, ViViT [3] and Timesformer [6] were the ﬁrst two works that successfully adopted a pure transformer architecture for video classiﬁcation, advancing the state of the art previously set by 3D CNNs. Multiscale processing in computer vision .”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “, Moments in Time [57] and Kinetics [32].”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “Multi-Moments in Time [43] 2019 YouTube 850 1 1M 3.”
Data and its (dis)contents: A survey of dataset development and use in machine learning research (2020, background): “Pipkin spent hundreds of hours watching the entirety of MIT’s ‘Moments in Time’ video dataset [Monfort et al., 2019].”
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures (2019, neutral): “The Moments in Time (MiT) dataset [16] is a large-scale video classiﬁcation dataset with more than 800K videos ( ∼ 3 seconds per video).”
CoCa: Contrastive Captioners are Image-Text Foundation Models (2022, methodology): “With a frozen-encoder, CoCa achieves 90.6% on ImageNet classification, 88.0%/88.5%/81.1% on Kinetics40/600/700 and 47.4% on Moments-in-Time.”
ViViT: A Video Vision Transformer (2021, methodology): “Moments in Time [45] consists of 800 000, 3-second YouTube clips that capture the gist of a dynamic scene involving animals, objects, people, or natural phenomena.”
Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction. (2018, methodology): “During training, the class labels of the videos in the two datasets (Kinetics [20] and MT [31]) are excluded and videos applied with the four rotations are used to train the 3DRotNet.”
Understanding Human Hands in Contact at Internet Scale (2020, neutral): “Most recently, [24] (5.6K)D None / Captions YouTube Moments [26] 34D, 17H Vid.”
Unsupervised Learning From Video With Deep Neural Embeddings (2019, background): “Improvements in GPU-based computing frameworks have ameliorated some of the most pressing computational issues [22], while several efforts have assembled larger-scale curated video benchmarks [20, 13, 1].”
MoViNets: Mobile Video Networks for Efficient Video Recognition (2021, methodology): “Results on Charades provide evidence that our models are also capable of sophisticated temporal understanding, as these videos can have longer duration clips than what is seen in Kinetics and Moments in Time.”
Attention Bottlenecks for Multimodal Fusion (2021, background): “In this section we report results on 2 additional datasets, Moments in Time [47] and Kinetics [35].”
Phenaki: Variable Length Video Generation From Open Domain Textual Description (2022, methodology): “The MiT dataset, in contrast to other publicly available video datasets, is a high quality balanced dataset with high coverage and density of verbs depicting moments of a few seconds [29].”
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey (2019, background): “unding box, segmentation mask YFCC100M [114] Image/Video 100 million media data 7 — Hashtags SceneNet RGB-D [115] Video 5 million images 3 13 Depth, Instance Segmentation, Optical Flow Moment-in-Time [116] Video 1 million 3-second videos 7 339 Video category class Kinetics [17] Video 0:5 million 10-”
Only Time Can Tell: Discovering Temporal Data for Temporal Modeling (2019, methodology): “These datasets are often created by first deciding on a taxonomy of classes and then looking for videos containing instances of such classes, or recording them [21].”