Is Space-Time Attention All You Need for Video Understanding?

By Gedas Bertasius, Heng Wang, Lorenzo Torresani. 2021

3,167 citations recorded by the source corpus.

We present a convolution-free approach to video classification built exclusively on self-attention over space and time. Our method, named"TimeSformer,"adapts the standard Transformer architecture to video by enabling spatiotemporal feature learning directly from a sequence of frame-level patches. Our experimental study compares different self-attention schemes and suggests that"divided attention,"where temporal attention and spatial attention are separately applied within each block, leads to the best video classification accuracy among the design choices considered. Despite the radically new design, TimeSformer achieves state-of-the-art results on several action recognition benchmarks, including the best reported accuracy on Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks, our model is faster to train, it can achieve dramatically higher test efficiency (at a small drop in accuracy), and it can also be applied to much longer video clips (over one minute long). Code and models are available at: https://github.com/facebookresearch/TimeSformer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jianfeng Gao
  2. Mohit Bansal
  3. Quoc V. Le
  4. Yoshua Bengio
  5. Trevor Darrell
  6. Dacheng Tao
  7. Percy Liang
  8. Sergey Levine

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Ming-Hsuan Yang
  4. Trevor Darrell
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Cordelia Schmid

In Model Architectures

  1. Christoph Feichtenhofer
  2. Luc Van Gool
  3. Lu Yuan
  4. Hongsheng Li
  5. Jianfeng Gao
  6. Yu Qiao
  7. Shuicheng Yan
  8. Jianwei Yang

In Human-Computer Interaction

  1. Hossein Rahmani
  2. Yu Qiao
  3. Jun Liu
  4. Cordelia Schmid
  5. Limin Wang
  6. Andrew Zisserman
  7. Ivan Laptev
  8. Kristen Grauman

Citations in the corpus