Attend and Interact: Higher-Order Object Interactions for Video Understanding

By Chih-Yao Ma, Asim Kadav, Iain Melvin, Zsolt Kira, Ghassan Al-Regib, Hans Peter Graf. 2017

149 citations recorded by the source corpus.

Human actions often involve complex interactions across several inter-related objects in the scene. However, existing approaches to fine-grained video understanding or visual relationship detection often rely on single object representation or pairwise object relationships. Furthermore, learning interactions across multiple objects in hundreds of frames for video is computationally infeasible and performance may suffer since a large combinatorial space has to be modeled. In this paper, we propose to efficiently learn higher-order interactions between arbitrary subgroups of objects for fine-grained video understanding. We demonstrate that modeling object interactions significantly improves accuracy for both action recognition and video captioning, while saving more than 3-times the computation over traditional pairwise relationships. The proposed method is validated on two large-scale datasets: Kinetics and ActivityNet Captions. Our SINet and SINet-Caption achieve state-of-the-art performances on both datasets even though the videos are sampled at a maximum of 1 FPS. To the best of our knowledge, this is the first work modeling object interactions on open domain large-scale video datasets, and we additionally model higher-order object interactions which improves the performance with low computational costs.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Set Transformer (2018, methodology): “Similarly to our work, Ma et al. (2018) use the Transformer to model interactions between the objects in a video.”
Grounded Video Description (2018, methodology): “Then, in the second stage, they either attend to the object regions dynamically [16, 38, 1] or classify the regions into labels and fill into pre-defined/generated sentence templates [15, 5, 12].”
Long-Term Feature Banks for Detailed Video Understanding (2018, methodology): “To model finer-grained interactions, a growing line of work leverages pre-computed object proposals [52] or detections [4,30], and models their co-occurrence [30,43, 52], temporal order [4], or spatial arrangement [52] within a short clip.”
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation (2019, background): “…of vision tasks like image captioning (Hu et al., 2016; Rohrbach et al., 2016; Lu et al., 2018), visual question answering (Lu et al., 2016b; Agrawal et al., 2018), relationship detection (Lu et al., 2016a; Ma et al., 2018) and referral expression (Nagaraja et al., 2016; Gavrilyuk et al., 2018).”
Ego-Topo: Environment Affordances From Egocentric Video (2020, unknown): “Graph-based methods encode relationships between detected objects: nodes are objects or actors, and edges specify their spatio-temporal layout or semantic relationships ( e.g ., is-holding) [68, 4, 46, 72].”
Videos as Space-Time Region Graphs (2018, background): “In the case of action recognition, a lot of effort has been made on modeling pairwise human-object and object-object relationships [55,56,57].”
Spatio-Temporal Graph for Video Captioning With Knowledge Distillation (2020, background): “Some works try operating on object features instead, but they either ignore cross-object interaction [49], or object transformation over time [27, 51].”
MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment (2018, background): “Recently, visual relations have been combined with deep neural networks in areas such as object recognition [21, 11], visual question answering [40] and action recognition [31, 34].”
Multi-interaction Network with Object Relation for Video Question Answering (2019, background): “[19] propose to efficiently learn higher-order interactions between arbitrary subgroups of objects for fine-grained video understanding.”
An Empirical Study on Leveraging Scene Graphs for Visual Question Answering (2019, background): “As an abstraction of images, scene graphs have been shown to improve image retrieval [31], generation [34], captioning [1, 22, 72], video understanding [49, 63], and human object interaction [36].”
Temporal Attentive Alignment for Large-Scale Video Domain Adaptation (2019, background): “Moreover, some works extract temporal dependencies between frames for video tasks by utilizing recurrent neural networks (RNNs) [6], attention [27, 29] and relation modules [56].”
Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction (2018, background): “[12] implicitly model the higher-order interactions among object region proposals, using groups and subgroups rather than just pairwise interactions.”