Is Space-Time Attention All You Need for Video Understanding?
By Gedas Bertasius, Heng Wang, Lorenzo Torresani. 2021
3,167 citations recorded by the source corpus.
We present a convolution-free approach to video classification built exclusively on self-attention over space and time. Our method, named"TimeSformer,"adapts the standard Transformer architecture to video by enabling spatiotemporal feature learning directly from a sequence of frame-level patches. Our experimental study compares different self-attention schemes and suggests that"divided attention,"where temporal attention and spatial attention are separately applied within each block, leads to the best video classification accuracy among the design choices considered. Despite the radically new design, TimeSformer achieves state-of-the-art results on several action recognition benchmarks, including the best reported accuracy on Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks, our model is faster to train, it can achieve dramatically higher test efficiency (at a small drop in accuracy), and it can also be applied to much longer video clips (over one minute long). Code and models are available at: https://github.com/facebookresearch/TimeSformer.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Human-Computer Interaction
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Quoc V. Le
- Yoshua Bengio
- Trevor Darrell
- Dacheng Tao
- Percy Liang
- Sergey Levine
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Cordelia Schmid
In Model Architectures
- Christoph Feichtenhofer
- Luc Van Gool
- Lu Yuan
- Hongsheng Li
- Jianfeng Gao
- Yu Qiao
- Shuicheng Yan
- Jianwei Yang
In Human-Computer Interaction
- Hossein Rahmani
- Yu Qiao
- Jun Liu
- Cordelia Schmid
- Limin Wang
- Andrew Zisserman
- Ivan Laptev
- Kristen Grauman
Citations in the corpus
- All in One: Exploring Unified Video-Language Pre-Training (2022, methodology): “For example, by simply adding a temporal attention layer to each block of the Transformer, the model becomes a normal Timesformer [5] with parameters increased from 86M to 121.7M (an increase of 42%).”
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, methodology): “Further performance improvement can be achieved if the BLIP model is used to initialize a video-language model with temporal modeling (e.g. replace our ViT with a TimeSformer (Bertasius et al., 2021)) and finetuned on video data.”
- Multiview Transformers for Video Recognition (2022, methodology): “We report accuracies when averaging predictions across multiple spatio-temporal crops, as standard practice [3, 6, 10, 23].”
- MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “[9] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.”
- FLAVA: A Foundational Language And Vision Alignment Model (2021, background): “The self-supervised pretraining paradigm has significantly advanced the state of the art across various domains, from natural language processing [6, 17–19, 23, 24, 28, 30, 61, 68, 73, 83–85], to computer vision [2, 5, 8, 12, 31, 33, 37, 59, 75, 104, 117], to speech recognition [4, 22, 42, 67, 119] ”
- VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, neutral): “In terms of model architecture, instead of naive mean pooling or concatenation over a sequence of individual frame features, VIOLET contains Video Swin Transformer that models video temporal explicitly for VidL learning [21,22].”
- Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, methodology): “Following the protocol in ViT and Timesformer [6], the input video clip is divided into M × N non-overlapping spatio-temporal patches of size P × P , where N = HW/P 2 .”
- Transformers in Vision: A Survey (2021, background): “TimeSFormer [18] extends ViTs [57] to videos, by considering the video as a sequence of patches extracted from individual frames.”
- VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, methodology): “On all datasets, we achieve higher accuracy than previous works including TimeSFormer [10], a recent effort in fine-tuning the ViT checkpoints obtained by supervised pretraining on image classification.”
- Multiscale Vision Transformers (2021, methodology): “Concurrent vision-transformer based methods [78,6,1] require over 5× more computation and large-scale external pre-training on ImageNet-21K (IN-21K), to achieve equivalent MViT accuracy.”
- ViViT: A Video Vision Transformer (2021, methodology): “We surpass the previous CNN-based state-of-the-art us-ing ViViT-L/16x2 Factorised Encoder (FE) pretrained on ImageNet, and also outperform [4] who concurrently proposed a pure-transformer architecture.”
- CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval (2021, background): “The transformer based video encoder is still in its early stage for action classification (Bertasius et al., 2021; Arnab et al., 2021).”
- Video Diffusion Models (2022, background): “The use of factorized space-time attention is known to be a good choice in video transformers for its computational efficiency [2, 5, 21].”
- XCiT: Cross-Covariance Image Transformers (2021, background): “Promising results have been obtained for other vision tasks, including image retrieval [22], object detection and semantic segmentation [44, 70, 81, 83], as well as video understanding [2, 7, 23].”
- GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, methodology): “Inspired by the success of Transformers in NLP [20,76], the Vision Transformer (ViT) [22] was recently proposed and has been successfully applied to multiple computer vision tasks, including image classification [44, 72, 73, 86], object detection [44, 79, 88], semantic segmentation [44, 82, 91] and ”
- MoViNets: Mobile Video Networks for Efficient Video Recognition (2021, methodology): “Even when compared to fully Transformer [66] models like TimeSformerHR [2], MoViNet-A6 outperforms it by 1% accuracy and using 40% of the FLOPs.”
- Prompting Visual-Language Models for Efficient Video Understanding (2021, background): “In the last decade, research on developing effective architectures has gone through rapid developments, from two-stream networks [15, 61, 66] to more recent single stream RGB networks [2, 8, 13, 14, 65, 70].”