Spatial-Aware Object Embeddings for Zero-Shot Localization and Classification of Actions

By Pascal Mettes, Cees G. M. Snoek. 2017

94 citations recorded by the source corpus.

We aim for zero-shot localization and classification of human actions in video. Where traditional approaches rely on global attribute or object classification scores for their zero-shot knowledge transfer, our main contribution is a spatial-aware object embedding. To arrive at spatial awareness, we build our embedding on top of freely available actor and object detectors. Relevance of objects is determined in a word embedding space and further enforced with estimated spatial preferences. Besides local object awareness, we also embed global object awareness into our embedding to maximize actor and object interaction. Finally, we exploit the object positions and sizes in the spatial-aware embedding to demonstrate a new spatiotemporal action retrieval scenario with composite queries. Action localization and classification experiments on four contemporary action video datasets support our proposal. Apart from state-of-the-art results in the zero-shot localization and classification settings, our spatial-aware embedding is even competitive with recent supervised action localization alternatives.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Pichao Wang
2. Wanqing Li
3. Jun Liu
4. Philip O. Ogunbona
5. Jun Wan
6. Dima Damen
7. Sergio Escalera
8. Limin Wang

Citations in the corpus

• RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “[92] proposed a spatial-aware object embedding for zero-shot action localization and classification.”
• Action Modifiers: Learning From Adverbs in Instructional Videos (2019, background): “Several problems are at the intersection between language and video: captioning [24, 38, 55, 59], retrieval [9, 16, 21, 31, 33, 52, 54] and visual question answering [15, 56, 57, 61].”
• A Survey of Zero-Shot Learning (2019, unknown): “Finally, zero-shot learning is used for action localization [65, 101] (i.e., localizing actions coming from the unseen classes), event recounting [49] (i.e., recounting unseen events), and description generation [54] (i.e., generating text descriptions of unseen activities).”
• Learning Spatiotemporal Features via Video and Text Pair Discrimination (2020, methodology): “Following [34], we select different classes for 10 times and average their accuracies for testing except the class number is 101.”