Asynchronous Temporal Fields for Action Recognition

By Gunnar A. Sigurdsson, Santosh Kumar Divvala, Ali Farhadi, Abhinav Kumar Gupta. 2016

173 citations recorded by the source corpus.

Actions are more than just movements and trajectories: we cook to eat and we hold a cup to drink from it. A thorough understanding of videos requires going beyond appearance modeling and necessitates reasoning about the sequence of activities, as well as the higher-level constructs such as intentions. But how do we model and reason about these? We propose a fully-connected temporal CRF model for reasoning over various aspects of activities that includes objects, actions, and intentions, where the potentials are predicted by a deep network. End-to-end training of such structured models is a challenging endeavor: For inference and learning we need to construct mini-batches consisting of whole videos, leading to mini-batches with only a few videos. This causes high-correlation between data points leading to breakdown of the backprop algorithm. To address this challenge, we present an asynchronous variational inference method that allows efficient end-to-end training. Our method achieves a classification mAP of 22.4% on the Charades [42] benchmark, outperforming the state-of-the-art (17.2% mAP), and offers equal gains on the task of temporal localization.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Time-Series & Forecasting

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Time-Series & Forecasting

Citations in the corpus

Non-local Neural Networks (2017, neutral): “8 Asyn-TF [43] RGB + flow 22.”
SlowFast Networks for Video Recognition (2018, neutral): “9 N/A Asyn-TF, VGG16 [42] ImageNet 22.”
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures (2019, neutral): “6 Asyn-TF (Sigurdsson et al., 2017) UCF101 RGB+Flow 22.”
Video Representation Learning Using Discriminative Pooling (2018, methodology): “HMDB-51 (accuracy over 3 splits) Method Accuracy Temporal segment networks[54] 69.4% AdaScan[19] 54.9% AdaScan + IDT + C3D[19] 66.9% ST ResNet[10] 66.4% ST ResNet + IDT[10] 70.3% ST Multiplier Network[11] 68.9% ST Multiplier Network + IDT[11] 72.2% Two-stream I3D[5] 66.4% Two-stream I3D+ (Kinetics 3”
Temporal Relational Reasoning in Videos (2017, methodology): “The TRN-equipped networks also obtain competitive results on activity classification in the Charades dataset [22], outperforming the Flow+RGB ensemble models [20, 22] using only sparsely sampled RGB frames.”
Video Action Transformer Network (2018, background): “Another thread of work has focused on temporal action detection [37, 38, 49]; however, it does not tackle the tasks of person detection or personaction attribution.”
Long-Term Feature Banks for Detailed Video Understanding (2018, neutral): “[39] (from [37]) VGG16 RGB+Flow 18.”
A Survey on Bayesian Deep Learning (2016, methodology): “Computer Vision Asynchronous Temporal Fields [102] LV ✓ Attend, Infer, Repeat (AIR) [20] LV ✓ Fast AIR [105] LV ✓ Sequential AIR [60] LV ✓”
Compressed Video Action Recognition (2017, methodology): “[31], our method achieves the best performance.”
Diagnosing Error in Temporal Action Detectors (2018, background): “For example, multiple works have tried to capture the temporal context along the video [8,12,24,35,45] as a proxy for the localization of the actions.”
R-C3D: Region Convolutional 3D Network for Temporal Activity Detection (2017, methodology): “We report the results using the same evaluation metric as in [25]. mAP standard post-process Random [25] 4.2 4.2 RGB [25] 7.7 8.8 Two-Stream [25] 7.7 10.0 Two-Stream+LSTM [25] 8.3 8.8 Sigurdsson et al. [25] 9.6 12.1 R-C3D (ours) 12.4 12.7 this as a future task.”
Hidden Two-Stream Convolutional Networks for Action Recognition (2017, methodology): “We have moved from manually designed features [1], [2], [3], [4], [5] to learned convolutional neural network (CNN) features [6], [7], [8], [9]; from encoding appearance information to encoding motion information [10], [11], [12], [13]; and from learning local features to learning global video featu”
Video Relationship Reasoning Using Gated Spatio-Temporal Energy Graph (2019, methodology): “However, the exact inference in a fully connected CRF is often computationally intractable even with variables enumeration or elimination [13].”
What Actions are Needed for Understanding Human Actions in Videos? (2017, methodology): “We find that instantaneous motion affects algorithms differently, for example, more temporal reasoning seems to help (IDT [37], TFields [30]) algorithms be robust to motion.”
D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation (2019, background): “This has led to the development of action segmentation approaches [23, 34, 39] that aim to label every frame in the video and not just to classify trimmed video clips.”
Future Event Prediction: If and When (2019, background): “[18] propose a fully-connected temporal CRF model, which exploits intent information to predict in each frame the action being performed.”
Timeception for Complex Action Recognition (2018, methodology): “6 [33] Two-stream + LSTM RGB + Flow 17.”