Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding

By Gunnar A. Sigurdsson, Gul Varol, X. Wang, Ali Farhadi, Ivan Laptev, Abhinav Kumar Gupta. 2016

1,422 citations recorded by the source corpus.

Computer vision has a great potential to help our daily lives by searching for lost keys, watering flowers or reminding us to take a pill. To succeed with such tasks, computer vision methods need to be trained from real and diverse examples of our daily dynamic scenes. While most of such scenes are not particularly exciting, they typically do not appear on YouTube, in movies or TV broadcasts. So how do we collect sufficiently many diverse but boring samples representing our lives? We propose a novel Hollywood in Homes approach to collect such data. Instead of shooting videos in the lab, we ensure diversity by distributing and crowdsourcing the whole process of video creation from script writing to video recording and annotation. Following this procedure we collect a new dataset, Charades, with hundreds of people recording videos in their own homes, acting out casual everyday activities. The dataset is composed of 9,848 annotated videos with an average length of 30 s, showing activities of 267 people from three continents. Each video is annotated by multiple free-text descriptions, action labels, action intervals and classes of interacted objects. In total, Charades provides 27,847 video descriptions, 66,500 temporally localized intervals for 157 action classes and 41,104 labels for 46 object classes. Using this rich data, we evaluate and provide baseline results for several tasks including action recognition and automatic description generation. We believe that the realism, diversity, and casual nature of this dataset will present unique challenges and new opportunities for computer vision community.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Non-local Neural Networks (2017, background): “Charades [44] is a video dataset with ∼8k training, ∼1.”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, neutral): “Charades [48] 10k 16k 10,000 82h Home 2016 MSR-VTT [58] 10k 200k 7,180 40h Youtube 2016 YouCook2 [67] 14k 14k 2,000 176h Youtube 2018 EPIC-KITCHENS [7] 40k 40k 432 55h Home 2018 DiDeMo [15] 27k 41k 10,464 87h Flickr 2017 M-VAD [52] 49k 56k 92 84h Movies 2015 MPII-MD [43] 69k 68k 94 41h Movies 2015 A”
Joint Discovery of Object States and Manipulation Actions (2017, background): “in instructional videos [3, 32, 37] or while performing daily activities [38].”
Towards Automatic Learning of Procedures From Web Instructional Videos (2017, background): “They provide limited or no annotations on procedure segment boundaries or descriptions (Sigurdsson et al. 2016).”
EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations (2022, background): “8 [37] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta.”
Dense-Captioning Events in Videos (2017, background): “ [39] movie 68k 4s 68,375 X - - - M-VAD [47] movie 49k 6s 55,904 X - - - MSR-VTT [55] open 10k 20s 200,000 X - - - MSVD [6] human 2k 10s 70,028 X - - - YouCook [7] cooking 88 - 2,688 X - - - Charades [43] human 10k 30s 16,129 X - - - KITTI [12] driving 21 30s 520 X X - - TACoS [36] cooking 127 360s ”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, background): “However, the recent creation of Sports-1M [5], Kinetics [6], Something-something [7], ActivityNet [8], Charades [9], etc. has partially removed that impediment.”
AVA: A Video Dataset of Spatio-Temporally Localized Atomic Visual Actions (2017, background): “ActivityNet [6], THUMOS [16], MultiTHUMOS [36] and Charades [32] use large numbers of untrimmed videos, each containing multiple actions, obtained either from YouTube (ActivityNet, THUMOS, MultiTHUMOS) or from crowdsourced actors (Charades).”
SlowFast Networks for Video Recognition (2018, methodology): “For the action classiﬁcation experiments, presented in this section we consider the widely used Kinetics-400 [30], the recent Kinetics-600 [3], and Charades [43].”
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures (2019, positive): “We first test on the popular Charades dataset (Sigurdsson et al., 2016) which is unique in the activity recognition domain as it contains long sequences.”
The “Something Something” Video Database for Learning and Evaluating Visual Common Sense (2017, background): “In contrast to the dataset described in [28] we focus here on basic, physical concepts rather than on higher-level human activities.”
TSM: Temporal Shift Module for Efficient Video Understanding (2018, background): “As pointed in [58, 53], datasets like Something-Something (V1&V2) [14], Charades [38], and Jester [1] are more focused on modeling the temporal relationships , while UCF101 [40], HMDB51 [26], and Kinetics [25] are less sensitive to temporal relationships.”