Object Level Visual Reasoning in Videos

By Fabien Baradel, Natalia Neverova, Christian Wolf, Julien Mille, Greg Mori. 2018

173 citations recorded by the source corpus.

Human activity recognition is typically addressed by detecting key concepts like global and local motion, features related to object classes present in the scene, as well as features related to the global context. The next open challenges in activity recognition require a level of understanding that pushes beyond this and call for models with capabilities for fine distinction and detailed comprehension of interactions between actors and objects in a scene. We propose a model capable of learning to reason about semantically meaningful spatio-temporal interactions in videos. The key to our approach is a choice of performing this reasoning at the object level through the integration of state of the art object detection networks. This allows the model to learn detailed spatial interactions that exist at a semantic, object-interaction relevant level. We evaluate our method on three standard datasets (Twenty-BN Something-Something, VLOG and EPIC Kitchens) and achieve state of the art results on all of them. Finally, we show visualizations of the interactions learned by the model, which illustrate object classes and their interactions corresponding to different activity classes.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations (2022, background): “3, 8 [4] Fabien Baradel, Natalia Neverova, Christian Wolf, Julien Mille, and Greg Mori.”
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (2019, background): “First, constructing 3D object-based representations for realistic scenes needs further exploration (Anderson et al., 2018; Baradel et al., 2018).”
Video Action Transformer Network (2018, neutral): “Unlike previous works [3], our model learns to do so without needing explicit object detections.”
Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition (2019, methodology): “For our ablation studies, following [5] we construct a validation set of unseen kitchen environments.”
Long-Term Feature Banks for Detailed Video Understanding (2018, methodology): “To model finer-grained interactions, a growing line of work leverages pre-computed object proposals [52] or detections [4,30], and models their co-occurrence [30,43, 52], temporal order [4], or spatial arrangement [52] within a short clip.”
Understanding Human Hands in Contact at Internet Scale (2020, background): “Recent work has largely taken the approach of recognizing verb-noun pairs [17, 9, 10, 19, 6].”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, background): “To assist more complicated action understanding, another growing set of research explores structural information in videos including temporal ordering [51, 88], object localization [4, 25, 32, 53, 74, 76], and implicit interactions between objects [4, 53].”
COPHY: Counterfactual Learning of Physical Dynamics (2019, methodology): “Following the same trend, (Baradel et al., 2018) estimate object relations from semantically well deﬁned entities using instance segmentation predictions for video understanding.”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, methodology): “RNNs have been successfully used before to recognize actions [2, 77].”
H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions (2019, methodology): “To do so, instead of directly feeding the hand and object poses as input to the temporal module, inspired by [1, 56], we model dependencies between hands and objects with a composite learned function and only then give the resulting mapping as input to RNN:”
Egocentric Hand Track and Object-Based Human Action Recognition (2019, background): “End-to-end methods also include [21], where in order to recognize actions a network is trained on pairs of frames and is jointly optimized for the training objectives of action recognition, object segmentation and interframe object interactions and their temporal association with recurrent networks.”
Ego-Topo: Environment Affordances From Egocentric Video (2020, unknown): “Graph-based methods encode relationships between detected objects: nodes are objects or actors, and edges specify their spatio-temporal layout or semantic relationships ( e.g ., is-holding) [68, 4, 46, 72].”
Unified Graph Structured Models for Video Understanding (2021, background): “A similar idea of using explicit object representations in video has also been employed by [3] and [60], though not for spatio-temporal action recognition.”
What Makes Training Multi-Modal Classification Networks Hard? (2019, methodology): “For ablations, following [8], we construct a validation set of unseen kitchen environments.”
Grouped Spatial-Temporal Aggregation for Efficient Action Recognition (2019, methodology): “Results on validation set of Epic-Kitchen verb classification tasks using the same split as in [1]”
The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines (2020, methodology): “Of these, recent research has explored using EPIC-KITCHENS for: video object reasoning and detection [67], [68], action retrieval [69], visual learning of novel words [57], unsupervised domain adaptation [70] and learning environmental affordances [71].”
Audiovisual SlowFast Networks for Video Recognition (2020, unknown): “We use the train/val split following [6].”
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (2021, background): “These include attention mechanisms from gaze and motor attention [47, 38, 42], active object detection [20, 7, 15, 70], and hands in contact [66, 60, 33].”