Recognizing action at a distance

By Alexei A. Efros, Alexander C. Berg, Greg Mori, Jitendra Malik. Proceedings Ninth IEEE International Conference on Computer Vision, 2003

1,392 citations recorded by the source corpus.

Our goal is to recognize human action at a distance, at resolutions where a whole person may be, say, 30 pixels tall. We introduce a novel motion descriptor based on optical flow measurements in a spatiotemporal volume for each stabilized human figure, and an associated similarity measure to be used in a nearest-neighbor framework. Making use of noisy optical flow measurements is the key challenge, which is addressed by treating optical flow not as precise pixel displacements, but rather as a spatial pattern of noisy measurements which are carefully smoothed and aggregated to form our spatiotemporal motion descriptor. To classify the action being performed by a human figure in a query sequence, we retrieve nearest neighbor(s) from a database of stored, annotated video sequences. We can also use these retrieved exemplars to transfer 2D/3D skeletons onto the figures in the query sequence, as well as two forms of data-based action synthesis "do as I do" and "do as I say". Results are demonstrated on ballet, tennis as well as football datasets.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/804d86dd7ab3498266922244e73a88c1add5a6ab

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Everybody Dance Now (2018, methodology): “Early methods focused on creating new content by manipulating existing video footage [5, 12, 31].”
Recognition and Segmentation of 3-D Human Action Using HMM and Multi-class AdaBoost (2006, methodology): “While some approaches process the video images directly as spatio-temporal volumes [10] , it is common to first detect and track humans to infer their actions [3] [12].”
A survey on vision-based human action recognition (2010, methodology): “[27] with histograms of oriented line segments.”
Temporal segmentation and activity classification from first-person sensing (2009, background): “[7] recognize actions at a distance on a ballet, tennis, and a soccer dataset.”
Free viewpoint action recognition using motion history volumes (2006, positive): “Such descriptors are not invariant to viewpoint, which can be partially resolved by multiplying the number of action classes by the number of possible viewpoints [6], relative motion directions [5], and point correspondences [7,8].”
Cross-View Action Recognition via a Continuous Virtual Path (2013, background): “Recently, in the field of action representation, several strategies have been proposed by researchers to make action representation more discriminative, such as space-time pattern templates [28], 2D shape matching [16, 19, 27], optical flow patterns [5], trajectory-based representation [22], and spa”
On Space-Time Interest Points (2005, neutral): “Other approaches use spatio-temporal image cues such as optical flow (Black et al., 1997) or motion templates (Baumberg and Hogg, 1996; Efros et al., 2003 ).”
Segmentation and Recognition Using Structure from Motion Point Clouds (2008, background): “Optical flow has aided recognition of objects for static cameras [23], but forward ego-motion dominates the visual changes in our footage.”
Shape Matching and Object Recognition (2006, methodology): “We originally introduced geometric blur in [Berg and Malik, 2001], and these ideas have been applied in [Efros et al., 2003] (in a temporal instead of spatial domain) and [Berg et al.”
Recognizing human actions using multiple features (2008, background): “In optical flow based approaches ([20, 21]), the idea is to directly use the optical flow as a basis for deriving a representation that can be used for recognition.”
A Biologically Inspired System for Action Recognition (2007, background): “The other common class of approaches is based on the processing of spatio-temporal features, either global as in the case of low-resolution videos [33, 6, 2] or local for higher resolution images [24, 5, 7, 18].”
Learning human actions via information maximization (2008, methodology): “al.[3] extracted an optical flow field as a descriptor from the stabilized object ST volume, and computed the cross correlation between the model and the input optical flow descriptors.”
Human Detection Using Oriented Histograms of Flow and Appearance (2006, methodology): “[4] used appearance and flow features in an exemplar based detector for long shots of sports players, but quantitative performance results were not given.”
Tensor Canonical Correlation Analysis for Action Classification (2007, background): “Some recent work has analyzed human actions directly in the space-time volume without explicit motion estimation [1] [4] [8] [7].”
Exploring Weak Stabilization for Motion Feature Extraction (2013, background): “Significant progress has been made in static-image object detection over the past few years, in large part due to the improvement of lowlevel features coupled with classifiers such as SVMs [7] and boosting [26].”
Adaptive Appearance Rendering (2021, methodology): “Human body pose is a natural intermediate representation for this generation, and hence utilized in many previous methods for synthesizing human motion and video [1, 4, 29].”
LabelMe: Online Image Annotation and Applications (2010, background): “A similar database was collected containing various sports actions [10].”
Human Action Recognition in Videos Using Kinematic Features and Multiple Instance Learning (2010, neutral): “Note that work by [3] and [4] can also be categorized under a volume-based representation.”