Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points

By Fabien Baradel, Christian Wolf, Julien Mille, Graham W. Taylor. 2018

177 citations recorded by the source corpus.

We propose a method for human activity recognition from RGB data that does not rely on any pose information during test time, and does not explicitly calculate pose information internally. Instead, a visual attention module learns to predict glimpse sequences in each frame. These glimpses correspond to interest points in the scene that are relevant to the classified activities. No spatial coherence is forced on the glimpse locations, which gives the attention module liberty to explore different points at each frame and better optimize the process of scrutinizing visual information. Tracking and sequentially integrating this kind of unstructured data is a challenge, which we address by separating the set of glimpses from a set of recurrent tracking/recognition workers. These workers receive glimpses, jointly performing subsequent motion tracking and activity prediction. The glimpses are soft-assigned to the workers, optimizing coherence of the assignments in space, time and feature space using an external memory module. No hard decisions are taken, i.e. each glimpse point is assigned to all existing workers, albeit with different importance. Our methods outperform the state-of-the-art on the largest human activity recognition dataset available to-date, NTU RGB+D, and on the Northwestern-UCLA Multiview Action 3D Dataset.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “Speciﬁcally, many of them have been evaluated based on the preliminary version [47] of our dataset, or pre-trained on it for transfer learning for other tasks [43], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [74], [75], [76], [77], [78], [79].”
2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning (2018, methodology): “Action recognition is improved by attention models that focus on body parts [4] and two-stream networks can be used to merge both RGB images and the costly optical flow maps [14].”
Object Level Visual Reasoning in Videos (2018, background): “In activity recognition, they have gained some traction in recent years, either as soft-attention on articulated pose (joints) [36], on feature map cells [34,39], on time [45] or on parts in raw RGB input through diﬀerentiable crops [3].”
Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition (2019, methodology): “During testing, similar to [2], we randomly create 5 new sequences in the similar manner and the mean score is used to predict the class.”
MMTM: Multimodal Transfer Module for CNN Fusion (2019, methodology): “Input Modalities Accuracy HCN [53] Pose 85.24 Infalated Resnet-50 [76] RGB 83.91 Two Stream [14] RGB+Pose 88.60 GMU [23] RGB+Pose 85.80 CentralNet [18] RGB+Pose 89.36 MFAS [17] RGB+Pose 90.04 Ours RGB+Pose 90.11”