ActionFormer: Localizing Moments of Actions with Transformers

By Chen-Lin Zhang, Jianxin Wu, Yin Li. European Conference on Computer Vision, 2022

534 citations recorded by the source corpus.

Self-attention based Transformer models have demonstrated impressive results for image classification and object detection, and more recently for video understanding. Inspired by this success, we investigate the application of Transformer networks for temporal action localization in videos. To this end, we present ActionFormer -- a simple yet powerful model to identify actions in time and recognize their categories in a single shot, without using action proposals or relying on pre-defined anchor windows. ActionFormer combines a multiscale feature representation with local self-attention, and uses a light-weighted decoder to classify every moment in time and estimate the corresponding action boundaries. We show that this orchestrated design results in major improvements upon prior works. Without bells and whistles, ActionFormer achieves 71.0% mAP at tIoU=0.5 on THUMOS14, outperforming the best prior model by 14.1 absolute percentage points. Further, ActionFormer demonstrates strong results on ActivityNet 1.3 (36.6% average mAP) and EPIC-Kitchens 100 (+13.5% average mAP over prior works). Our code is available at http://github.com/happyharrycn/actionformer_release.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/58a3fedc03ab9f5908c077c115eff4c8d2d87660

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Prompting Visual-Language Models for Efficient Video Understanding (2021, background): “The one-stage paradigm [7, 36, 46, 59, 83, 85] combines localisation and classification, i.”
Perception Test: A Diagnostic Benchmark for Multimodal Video Models (2023, methodology): “Temporal action localisation: We obtained baseline results for temporal action localisation using ActionFormer [57] with different pretrained features: TSP video features from [5] extracted using a Resnet(2+1)D-34 model pre-trained on ActivityNet, MMV audio features from [2] extracted using an S3D…”
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, background): “Finally, we believe the sequence-to-sequence design of Vid2Seq has the potential to be extended to a wide range of other video tasks such as temporally-grounded video question answering [51, 56, 57] or temporal action localization [16, 67, 123].”
VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking (2023, methodology): “We use the ActionFormer [86] detection pipeline as our baseline method and replace its I3D feature with our pre-trained representation.”
InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, methodology): “We report the performance of the state-of-the-art TAL methods whose codes are publicly available, including ActionFormer [4] method for THUMOS-14, ActivityNet-v1.”