Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers

By Mandela Patrick, Dylan Campbell, Yuki M. Asano, Ishan Misra Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, João F. Henriques. 2021

372 citations recorded by the source corpus.

In video transformers, the time dimension is often treated in the same way as the two spatial dimensions. However, in a scene where objects or the camera may move, a physical point imaged at one location in frame $t$ may be entirely unrelated to what is found at that location in frame $t+k$. These temporal correspondences should be modeled to facilitate learning about dynamic scenes. To this end, we propose a new drop-in block for video transformers -- trajectory attention -- that aggregates information along implicitly determined motion paths. We additionally propose a new method to address the quadratic dependence of computation and memory on the input size, which is particularly important for high resolution or long videos. While these ideas are useful in a range of settings, we apply them to the specific task of video action recognition with a transformer model and obtain state-of-the-art results on the Kinetics, Something--Something V2, and Epic-Kitchens datasets. Code and models are available at: https://github.com/facebookresearch/Motionformer

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Multiview Transformers for Video Recognition (2022, methodology): “We improve substantially over ViViT-L-FE [3], which corresponds to a deeper single-view equivalent of our model by 2.6%, and also improve upon MFormer [51] by 0.4%.”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “[157] injected a trajectory attention block into the Transformer network to enhance the robustness of HAR in dynamic scenes.”
Earthformer: Exploring Space-Time Transformers for Earth System Forecasting (2022, methodology): “Beyond minimal adaptation from ViT, some recent work introduced more prior to the design of space-time transformers, including trajectory [29], multi-scale [23, 11] and multi-view [49].”
XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model (2022, methodology): “Note, similar prototypical approximations have been used in transformers [55,38].”
VidTr: Video Transformer Without Convolutions (2021, background): “Some more recent work [2, 4, 4, 16, 40, 42] further studied the multi-scale and different attention factorization methods.”
VideoMamba: State Space Model for Efficient Video Understanding (2024, positive): “We omit the initial normalization and the ﬁnal residual for simpliﬁcation. models [2,4,60,64,90] like TimeSformer [4] and ViViT [2] signiﬁcantly advanced the ﬁeld by capturing long-range dependencies, improving temporal relationship understanding.”
VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training (2022, background): “8 Motionformer [51] ViT-L 3 32 1185×10×3 382 80.”
Long-Short Temporal Contrastive Learning of Video Transformers (2021, methodology): “ViT models were also adapted to the video domain by introducing different forms of spatiotemporal self-attention [2, 3, 6, 49].”
UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning (2022, background): “…several prior works (Bertasius et al., 2021; Neimark et al., 2021; Sharir et al., 2021; Li et al., 2021b; Arnab et al., 2021; Bulat et al., 2021; Patrick et al., 2021; Zha et al., 2021) propose different variants for spatiotemporal learning, verifying the outstanding ability of the transformer…”
ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, background): “As temporal information is important for modeling the dynamics, a variety of motion learning techniques has been introduced [83, 33, 54].”
Expanding Language-Image Pretrained Models for General Video Recognition (2022, neutral): “7 1 × 3 2380 3 Mformer-HR [34] IN-21k 16 81.”