Actions ~ Transformations

By X. Wang, Ali Farhadi, Abhinav Kumar Gupta. 2015

242 citations recorded by the source corpus.

What defines an action like "kicking ball"? We argue that the true meaning of an action lies in the change or transformation an action brings to the environment. In this paper, we propose a novel representation for actions by modeling an action as a transformation which changes the state of the environment before the action happens (precondition) to the state after the action (effect). Motivated by recent advancements of video representation using deep learning, we design a Siamese network which models the action as a transformation on a high-level feature space. We show that our model gives improvements on standard action recognition datasets including UCF101 and HMDB51. More importantly, our approach is able to generalize beyond learned action categories and shows significant performance improvement on cross-category generalization on our new ACT dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Joint Discovery of Object States and Manipulation Actions (2017, methodology): “This observation has been used to design action models in [16, 17, 45].”
STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, background): “Different from spatio-temporal graphs [24, 41, 20], the hypergraph structure describes actions as hyperedges and instead of the frame-level subgraphs.”
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, methodology): “[27] first built an action list by combining previous video datasets [23, 29, 51, 3, 60], and then searched the videos from YouTube for candidates, which eventually were filtered by Amazon Mechanical Turkers.”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[223] Xiaolong Wang, Ali Farhadi, and Abhinav Gupta.”
The Kinetics Human Action Video Dataset (2017, methodology): “These sources include: (i) Action datasets – existing datasets like ActivityNet [3], HMDB [15], UCF101 [20], MPII Human Pose [2], ACT [25] have useful classes and a suitable sub set of these were used; (ii) Motion capture – there are a number of motion capture datasets which we looked through and ex”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, background): “Since then, many video classification methods follow the same multi-stream 2D CNN design, and have made improvements in terms of new representations [25,26], different backbone architecture [27,28,29,17], fusion of the streams [30,31,32,33] and exploiting richer temporal structures [34,35,36].”
Spatiotemporal Residual Networks for Video Action Recognition (2016, neutral): “gnition. Here, our gain of 4.8% further underlines the importance of our architectural choices. Method UCF101 HMDB51 Two-Stream ConvNet [20] 88.0% 59.4% Two-Stream+LSTM[18] 88.6% - Two-Stream (VGG16) [1,31] 91.4% 58.5% Transformations[31] 92.4% 62.0% Two-Stream Fusion[5] 92.5% 65.4% ST-ResNet* 93.4%”
A Closer Look at Spatiotemporal Convolutions for Action Recognition (2017, methodology): “Additional two-stream approaches include Temporal Segment Networks [39], Action Transformations [40], and Convolutional Fusion [10].”
Unsupervised Representation Learning by Sorting Sequences (2017, methodology): “ing methods. Dataset. The PASCAL VOC 2007 [7] dataset has 20 object classes and contains 5,011 images for training and 4,952 images for testing. We train our model using the UCF-101, HMDB-51, and ACT [40] datasets. For both tasks we use the same ﬁne-tuning strategy described in Kr¨ahenb ¨uhl et al. ”
Temporal Relational Reasoning in Videos (2017, methodology): “[23] uses a two-stream siamese network to learn the transformation matrix between two frames, then uses brute force search to infer the action category.”
Visual Semantic Planning Using Deep Successor Representations (2017, background): “One possible way is to learn these from still images or videos [12, 51, 52].”
Cross and Learn: Cross-Modal Self-Supervision (2018, methodology): “The ACT [39] dataset is composed of 11K video clips of people performing various actions.”
Asynchronous Temporal Fields for Action Recognition (2016, methodology): “Moving towards more expressive deep networks such as LSTM has become a popular method for encoding such temporal information [48, 4, 65, 50, 58, 41, 64].”
Trespassing the Boundaries: Labeling Temporal Bounds for Object Interactions in Egocentric Video (2017, background): “However, both approaches [9, 34] rely on manual annotations of actoms [9] or action segments [34], which are potentially as subjective as the temporal bounds of the actions themselves.”
Grasp2Vec: Learning Object Representations from Self-Supervised Grasping (2018, background): “al represent actions as the transformation from precondition to effect in the action recognition domain [13].”