Temporal Relational Reasoning in Videos

By Bolei Zhou, Alex Andonian, Antonio Torralba. 2017

1,140 citations recorded by the source corpus.

Temporal relational reasoning, the ability to link meaningful transformations of objects or entities over time, is a fundamental property of intelligent species. In this paper, we introduce an effective and interpretable network module, the Temporal Relation Network (TRN), designed to learn and reason about temporal dependencies between video frames at multiple time scales. We evaluate TRN-equipped networks on activity recognition tasks using three recent video datasets - Something-Something, Jester, and Charades - which fundamentally depend on temporal relational reasoning. Our results demonstrate that the proposed TRN gives convolutional neural networks a remarkable capacity to discover temporal relations in videos. Through only sparsely sampled video frames, TRN-equipped networks can accurately predict human-object interactions in the Something-Something dataset and identify various human gestures on the Jester dataset with very competitive performance. TRN-equipped networks also outperform two-stream networks and 3D convolution networks in recognizing daily activities in the Charades dataset. Further analyses show that the models learn intuitive and interpretable visual common sense knowledge in videos (Code and models are available at http://relation.csail.mit.edu/.).

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Weight Standardization (2019, methodology): “The codes are based on TRN [47] and then adapted to TSM.”
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “, TCN [24], TRN [62] and CRN [23]), sequential models (e.”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Notably, pre-training egocentric video models with third-person data [70,221,224,239] suffers from the sizeable domain mismatch [139,201].”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “of five state-of-the-art recognition models [66,67,68,69, 70] in addition to a random chance baseline.”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, methodology): “We also signiﬁcantly outperform the current state of the art method, which is the Multi-scale TRN of [39], improving top-1 accuracy from 33.6% to 42.0%.”
SlowFast Networks for Video Recognition (2018, methodology): “There are also related methods focusing on long-term filtering and pooling using temporal strides [52, 13, 55, 62], as well as decomposing the convolutions into separate 2D spatial and 1D temporal filters [12, 50, 61, 39].”
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures (2019, neutral): “Method modality mAP 2-Strm. [20] (from [18]) RGB+Flow 18.6 Asyn-TF [18] RGB+Flow 22.4 CoViAR [28] Compressed 21.9 MultiScale TRN [33] RGB 25.2 I3D [3] RGB 32.9 I3D [3] (from [25]) RGB 35.5 I3D-NL [25] RGB 37.5 STRG [26] RGB 39.7 LFB [27] RGB 42.5 SlowFast [6] RGB+RGB 45.”
Class-Incremental Learning for Action Recognition in Videos (2021, positive): “So, the proposed class-incremental learning framework employs a frame-based video representation method—Temporal Shift Module (TSM) [21], and reduces computational cost for training signiﬁcantly compared to 3D CNNs based on video volumes [2, 8, 9, 36, 37] and their variations [40, 47].”
Multiscale Vision Transformers (2021, background): “Incorporating downsampling, shift invariance, and shared weights, ConvNets are de-facto standard backbones for computer vision tasks for image [65, 62, 88, 90, 46, 12, 15, 34, 93, 81, 41] and video [87, 31, 11, 79, 69, 106, 96, 30, 105, 35, 29, 117, 57].”
TSM: Temporal Shift Module for Efficient Video Understanding (2018, methodology): “1 Two-stream TRNRGB+Flow [58] BNInception 8+8 36.”
ViViT: A Video Vision Transformer (2021, methodology): “It is thus analogous to CNN architectures such as [24, 34, 72, 86] which ﬁrst extract per-frame features, and then aggregate them into a ﬁnal representation before classifying them.”
Moments in Time Dataset: One Million Videos for Event Understanding (2018, methodology): “Temporal Relation Networks (TRN) [47] explicitly learn temporal dependencies between video segments that best characterize a particular action.”
Hierarchical Conditional Relation Networks for Video Question Answering (2020, methodology): “Within the second approach, the TRN network [48] demonstrates the role of temporal frame relations as an another important visual feature for video reasoning and VideoQA [16].”
VirtualHome: Simulating Household Activities Via Programs (2018, methodology): “We use DilatedNet to obtain the semantic segmentation of each frame and use the Temporal Relation Network [31] with 4-frame relations to predict the embedding of an instruction (action+object+object).”
Long-Term Feature Banks for Detailed Video Understanding (2018, methodology): “Temporal and relationship models include RNNs that model the evolution of video frames [7, 24, 27, 44, 57] and multilayer perceptrons that model ordered frame features [58].”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, background): “To assist more complicated action understanding, another growing set of research explores structural information in videos including temporal ordering [51, 88], object localization [4, 25, 32, 53, 74, 76], and implicit interactions between objects [4, 53].”
When Video Classification Meets Incremental Classes (2021, background): “For the task of video classification, existing datasets can be roughly divided into two groups: YouTube-type videos and crowd-sourced videos [58, 64].”
Unsupervised Learning From Video With Deep Neural Embeddings (2019, methodology): “A purely 2D approach is the Temporal Relational Network (TRN) [39], which processes aggregates of 2D convolutional features using MLP readouts.”
Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video (2020, background): “Zhou et al. [28] introduced Temporal Relation Network (TRN), a module capable of encoding temporal dependencies between video frames at multiple time scales.”