The “Something Something” Video Database for Learning and Evaluating Visual Common Sense

By Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, S. Westphal, Heuna Kim, V. Haenel, Ingo Fründ, P. Yianilos, Moritz Mueller-Freitag and 4 more. 2017

2,008 citations recorded by the source corpus.

Neural networks trained on datasets such as ImageNet have led to major advances in visual object classification. One obstacle that prevents networks from reasoning more deeply about complex scenes and situations, and from integrating visual knowledge with natural language, like humans do, is their lack of common sense knowledge about the physical world. Videos, unlike still images, contain a wealth of detailed information about the physical world. However, most labelled video datasets represent high-level concepts rather than detailed physical aspects about actions and scenes. In this work, we describe our ongoing collection of the “something-something” database of video prediction tasks whose solutions require a common sense understanding of the depicted situation. The database currently contains more than 100,000 videos across 174 classes, which are defined as caption-templates. We also describe the challenges in crowd-sourcing this data at scale.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Weight Standardization (2019, methodology): “…experiments, including image classiﬁcation on ImageNet dataset [35], object detection and instance segmentation on COCO dataset [22], video recognition on Something-SomethingV1 dataset [8], semantic image segmentation on PASCAL VOC [6], and point cloud classiﬁcation on ModelNet40 dataset [43].”
Multiview Transformers for Video Recognition (2022, background): “This dataset consists of class labels such as “move to left” and “pointing to right” [26].”
ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition (2021, background): “Something-Something [12] and EPIC-Kitchens [7] are video datasets collected by users with mobile and head-mounted cameras, respectively, but are focused on action recognition based on many examples and “action captions”.”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[90] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al.”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, neutral): “0k 3 :1 54 4,025 a, v, n Top-1/5 Acc. Kinetics-700 [41,42] 2019 YouTube 1806 1 650k 10:00:0 700 a Top-1/5 Err. Multi-Moments in Time [43] 2019 YouTube 850 1 1M 3:00:0 313 ML-a mAP Something-Something [44] 2018 Scripted 234 1 220k 3:81:1 174 a, n, c Top-1/5 Acc. AVA [33] 2018 Film 108 1380 410k 2:73:”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, methodology): “However, the recent creation of Sports-1M [5], Kinetics [6], Something-something [7], ActivityNet [8], Charades [9], etc. has partially removed that impediment.”
Video Action Recognition Via Neural Architecture Searching (2019, background): “On one hand, new and large-scale datasets, such as Kinetics [4], SomethingSomething [5], make great contribution to video action recognition.”
Class-Incremental Learning for Action Recognition in Videos (2021, neutral): “We evaluate the proposed framework on UCF101 [35], HMDB51 [19] and Something-Something V2 [12], which are the standard datasets for action recognition tasks.”
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset (2018, background): “In [15], short segments linked to interactions with 30 daily objects are collected by querying YouTube, while [10], [16] are scripted – subjects are requested to enact a crowd-sourced storyline [16] or a given action [10], which oftentimes results in less natural looking actions.”
TSM: Temporal Shift Module for Efficient Video Understanding (2018, methodology): “For example, on Something-Something-V1 dataset, our TSM achieves 25.1% better performance than the 2D baseline, even though the computation costs of the two methods are exactly the same.”
ViViT: A Video Vision Transformer (2021, methodology): “Something-Something v2 (SSv2) [26] contains 220 000 videos, with durations ranging from 2 to 6 seconds.”
Moments in Time Dataset: One Million Videos for Event Understanding (2018, background): “These include UCF101 [40], ActivityNet [8], Kinetics [25], Something-Something [14], AVA [15], and Charades [38].”
Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition (2019, methodology): “We report the hyper parameters when target dataset is Ki-netics in Tables 8 – 12, EPIC-Kitchens in Tables 14 – 16, and Something-something-v1 in Table 17.”
When Video Classification Meets Incremental Classes (2021, methodology): “We select two popular large-scale video classification datasets SomethingSomething V2 [15] (i.”
Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings (2019, background): “Recently, several large-scale datasets have been published for the task of ﬁne-grained action recognition [6, 12, 13, 31, 28].”