HMDB: A large video database for human motion recognition

By Hilde Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso A. Poggio, Thomas Serre. 2011

4,317 citations recorded by the source corpus.

With nearly one billion online videos viewed everyday, an emerging new frontier in computer vision research is recognition and search in video. While much effort has been devoted to the collection and annotation of large scalable static image datasets containing thousands of image categories, human action datasets lag far behind. Current action recognition databases contain on the order of ten different action categories collected under fairly controlled conditions. State-of-the-art performance on these datasets is now near ceiling and thus there is a need for the design and creation of new benchmarks. To address this issue we collected the largest action video database to-date with 51 action categories, which in total contain around 7,000 manually annotated clips extracted from a variety of sources ranging from digitized movies to YouTube. We use this database to evaluate the performance of two representative computer vision systems for action recognition and explore the robustness of these methods under various conditions such as camera motion, viewpoint, video quality and occlusion.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, methodology): “Four datasets are relevant for our study: HMDB51 [16], UCF101 [25], Kinetics400 [1] and IG65M [9] datasets.”
Learning Video Representations from Textual Web Supervision (2020, methodology): “We compare our method with other self-supervised and cross-modal approaches, showing that our method produces video representations which improve downstream performance by 8.9% on HMDB-51 [36] and 2.1% on UCF 101 [57] (Section 5).”
Self-Supervised MultiModal Versatile Networks (2020, methodology): “Some classification datasets have official splits (3 for UCF101/HMDB51 and 5 for ESC-50).”
UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild (2012, methodology): “Recently, web videos have been used in order to utilize unconstrained user-uploaded data to alleviate the second issue [6, 8, 9, 5].”
MERLOT: Multimodal Neural Script Knowledge Models (2021, neutral): “[61] and UCF-101 [103].”
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, methodology): “[28] first built an action list by combining previous video datasets [24, 31, 51, 3, 60], and then searched the videos from YouTube for candidates, which eventually were filtered by Amazon Mechanical Turkers.”
Long-term recurrent convolutional networks for visual recognition and description (2014, neutral): “RGB frames and one CNN trained on a stack of 10 ﬂow frames. When combining RGB and ﬂow by averaging softmax scores, results are comparable to state-of-the-art shallow models on UCF101 [25] and HMDB51 [51]. Results ﬂow as opposed to simply averaging scores. Alternatively, [1] and [2] propose learning”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “Several datasets have been collected to focus on action recognition, from [52,53] to recent large-scale ones [5,41,43,44,45,46], all offering a challenge with a held-out test set.”
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset (2017, methodology): “In this paper we aim to provide an answer to this question using the new Kinetics Human Action Video Dataset [16], which is two orders of magnitude larger than previous datasets, HMDB-51 [18] and UCF-101 [27].”
Two-Stream Convolutional Networks for Action Recognition in Videos (2014, methodology): “Recent improvements of trajectory-based hand-crafted representations include compensation of global (camera) motion [10, 16, 26], and the use of the Fisher vector encoding [22] (in [26]) or its deeper variant [23] (in [21]).”
The Kinetics Human Action Video Dataset (2017, methodology): “Clips Total Videos HMDB-51 [15] 2011 51 min 102 6,766 3,312 UCF-101 [20] 2012 101 min 101 13,320 2,500 ActivityNet-200 [3] 1: Statistics for recent human action recognition datasets.”
Contrastive Bidirectional Transformer for Temporal Representation Learning (2019, background): “Most of the recent efforts on learning deep representations for videos have been focusing on capturing the appearance and local motion pattern from short video clips [22, 13, 11], in an supervised [26, 5, 35] or self-supervised [15, 31] fashion.”
Dense-Captioning Events in Videos (2017, background): “ studying human actions with a black background. Since then, datasets like UCF101 [45], Sports 1M [21], Thumos 15 [15] have focused on studying actions in sports related internet videos while HMDB 51 [25] and Hollywood 2 [28] introduced a dataset of movie clips. Recently, ActivityNet [4] and Charade”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, methodology): “Next we conduct transfer learning experiments from Kinetics to other video classification datasets, namely HMDB-51 [44] and UCF-101 [45].”
Unsupervised Learning of Video Representations using LSTMs (2015, background): “The HMDB-51 dataset (Kuehne et al., 2011) contains 5100 videos belonging to 51 different action categories.”
Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding (2016, methodology): “In order to scale up the learning and the complexity of the data, recent approaches have instead tried collecting video datasets by downloading videos from Internet.”
YouTube-8M: A Large-Scale Video Classification Benchmark (2016, background): “…understanding domain where the community has quickly progressed from small, well-labeled datasets such as KTH [22], Hollywood 2 [23], Weizmann [5], with a few thousand video clips, to medium-scale datasets such as UCF101 [33], Thumos‘14 [16] and HMDB51 [21], with more than 50 action categories.”
Beyond short snippets: Deep networks for video classification (2015, background): “These approaches are able to aggregate local appearance and motion information using hand-crafted features such as Histogram of Oriented Gradients (HOG), Histogram of Optical Flow (HOF), Motion Boundary Histogram (MBH) around spatio-temporal interest points [17], in a dense grid [24] or around dense”