MoDeep: A Deep Learning Framework Using Motion Features for Human Pose Estimation

By Arjun Jain, Jonathan Tompson, Yann LeCun, C. Bregler. 2014

176 citations recorded by the source corpus.

In this work, we propose a novel and efficient method for articulated human pose estimation in videos using a convolutional network architecture, which incorporates both color and motion features. We propose a new human body pose dataset, FLIC-motion (This dataset can be downloaded from http://cs.nyu.edu/~ajain/accv2014/.), that extends the FLIC dataset [1] with additional motion features. We apply our architecture to this dataset and report significantly better performance than current state-of-the-art pose detection systems.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Recent advances in convolutional neural networks (2015, methodology): “[154] also incorporate RGB features and motion features to a multiresolution CNN architecture to further improve accuracy.”
C3D: Generic Features for Video Analysis (2014, methodology): “ConvNets have been applied to the problem of human pose estimation in both images [17] and videos [18].”
Efficient object localization using Convolutional Networks (2014, methodology): “The model of Toshev et al. [22] significantly output-performed state-of-art methods on the challenging ‘FLIC’ [20] dataset and was competitive on the ‘LSP’ [16] dataset.”
Flowing ConvNets for Human Pose Estimation in Videos (2015, methodology): “Further improvements may be obtained by using additional inputs for the spatial ConvNet, for example multiple RGB frames [24] or optical flow [18] – although prior work has shown little benefit from this so far.”
Recurrent Human Pose Estimation (2016, background): “An alternative, which it turns out might be better suited to ConvNets, is an indirect prediction by first regressing a heatmap over the image for each keypoint, and then obtaining the keypoint position as a mode in this heatmap [20], [27], [38], [37].”
Monocular 3D Human Pose Estimation Using Transfer Learning and Improved CNN Supervision (2016, methodology): “Approaches using part detectors and graphical models [1, 22, 23, 5, 12] were successively outperformed by 2D CNN-based approaches [32, 31, 79, 51, 82].”
Pose for Action - Action for Pose (2016, methodology): “For pose estimation in videos, [17] combines the heatmaps of body parts from multiple frames with optical flow information to leverage the temporal information in videos.”
Deep learning for visual understanding: A review (2016, background): “…applications, such as video surveillance, human behavior analysis, human-computer interaction (HCI), and is being extensively studied recently [192 – 195 , 205 Normally, human pose estimation involves multiple problems such as recognizing people in images, detecting and describing human body…”
Recurrent Network Models for Human Dynamics (2015, methodology): “We further compare our video labeler in a subset of 200 video sequences of around 50 frames each from the FlicMotion dataset of [14, 25] that we annotated densely in time with person bounding boxes.”
Joint Customer Pose and Orientation Estimation Using Deep Neural Network from Surveillance Camera (2016, background): “[7] fused image patches with optical flow to predict the pose in image sequence.”
Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “[82] designed a two-branch CNN framework to incorporate both color and motion features within frame pairs to build an expressive temporal-spatial model in HPE.”
Gesture recognition for human-robot collaboration: A review (2017, methodology): “Please cite this article in press as: Liu, H., Wang, L., Gesture recognition for human-robot collaboration: A review, International Journal of Industrial Ergonomics (2017), http://dx.doi.org/10.1016/j.ergon.2017.02.004 (cid:1) Non-wearable sensor and deep learning based gesture recognition:…”
Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in Video (2015, methodology): “It is a core component in the two-stream architecture described by Simonyan and Zisserman [23] and is used for human pose estimation [10], for global video descriptor learning [20] and for video captioning [28].”
Deep Multimodal Learning: A Survey on Recent Advances and Trends (2017, background): “Related subfields of research include human pose estimation [14] and semantic scene understanding [15].”
Monocular 3D Human Pose Estimation in the Wild Using Improved CNN Supervision (2016, background): “Deep CNN architectures have dramatically improved 2D pose estimation [25, 38], with even real-time solutions [69].”
Efficient ConvNet-based marker-less motion capture in general scenes with a low number of cameras (2015, methodology): “Toshev et al. [40] formulate the problem as a direct regression to joint location.”
Computer vision and deep learning techniques for pedestrian detection and tracking: A survey (2018, background): “For example, in [112] , it is shown that deep learning is able to successfully incorporate both RGB and motion features for the task of human body pose detection in video.”
TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 1 ModDrop (2018, unknown): “On the other hand, extending these models to problems involving the understanding of video content is still in its infancy, this idea having been explored only in a small number of recent works [9], [10], [11], [12].”
Stacked Hourglass Networks for Human Pose Estimation (2016, background): “There are variations to the pose estimation problem which include use of additional features such as depth or motion cues [27,28].”