Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs

By Jingwei Ji, Ranjay Krishna, Li Fei-Fei, Juan Carlos Niebles. 2019

437 citations recorded by the source corpus.

Action recognition has typically treated actions and activities as monolithic events that occur in videos. However, there is evidence from Cognitive Science and Neuroscience that people actively encode activities into consistent hierarchical part structures. However, in Computer Vision, few explorations on representations that encode event partonomies have been made. Inspired by evidence that the prototypical unit of an event is an action-object interaction, we introduce Action Genome, a representation that decomposes actions into spatio-temporal scene graphs. Action Genome captures changes between objects and their pairwise relationships while an action occurs. It contains 10K videos with 0.4M objects and 1.7M visual relationships annotated. With Action Genome, we extend an existing action recognition model by incorporating scene graphs as spatio-temporal feature banks to achieve better performance on the Charades dataset. Next, by decomposing and learning the temporal changes in visual relationships that result in an action, we demonstrate the utility of a hierarchical event decomposition by enabling few-shot action recognition, achieving 42.7% mAP using as few as 10 examples. Finally, we benchmark existing scene graph models on the new task of spatio-temporal scene graph prediction.

Subfields: Computer Vision and Pattern Recognition, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Graph Learning

Citations in the corpus

EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations (2022, background): “3 [19] Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles.”
Unified Graph Structured Models for Video Understanding (2021, methodology): “Both tasks are evaluated using the standard Recall@K metric [39, 23] (R@K) which measures the fraction of ground truth relationship triplets (subject-predicate-object) that appear in the top K scoring predicted triplets.”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, background): “CAD120++ [174] and Action Genome [131] are two video action reasoning datasets containing scenes of human daily life.”
Grounding Predicates through Actions (2021, background): “The Action Genome dataset [18] introduces spatio-temporal scene graphs with the purpose of using symbolic state changes across actions to improve action recognition in videos.”
Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning (2021, background): “Later, visual reasoning was extended to more complex dynamic videos (Lei et al., 2018; Fan et al., 2019; Li et al., 2020; 2019a; Huang et al., 2020).”
Image Scene Graph Generation (SGG) Benchmark (2021, background): “…for building more visually-grounded and compositional models for other computer vision and vision-language tasks, like image/text retrieval [9, 17], image cap-tioning [4, 20, 6], visual question answering [3, 5], action recognition [7], image generation [8, 14], and 3D scene understanding [2].”
Learning to Generate Scene Graph from Natural Language Supervision (2021, background): “Scene graph has emerged as a structured representation for many vision tasks, including action recognition [19], 3D scene understanding [1, 49], image generation and editing [21, 13], and vision-language tasks (e.”
AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning (2021, methodology): “The programs that create questions operate over Charades’ action annotations and Action Genome’s spatio-temporal scene graphs, which ground all objects with bounding boxes and actions with time stamps in the video [19, 45].”