Video Action Transformer Network

By Rohit Girdhar, João Carreira, Carl Doersch, Andrew Zisserman. 2018

783 citations recorded by the source corpus.

We introduce the Action Transformer model for recognizing and localizing human actions in video clips. We repurpose a Transformer-style architecture to aggregate features from the spatiotemporal context around the person whose actions we are trying to classify. We show that by using high-resolution, person-specific, class-agnostic queries, the model spontaneously learns to track individual people and to pick up on semantic context from the actions of others. Additionally its attention mechanism learns to emphasize hands and faces, which are often crucial to discriminate an action – all without explicit supervision other than boxes and class labels. We train and test our Action Transformer network on the Atomic Visual Actions (AVA) dataset, outperforming the state-of-the-art by a significant margin using only raw RGB frames as input.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Scaling Autoregressive Video Models (2019, methodology): “The conceptual simplicity of our model is in line with recent approaches to video classification that models videos by means of 3D convolutions [5, 43] or spatiotemporal self-attention [15].”
Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, background): “Another similar study for action recognition is the video action transformer network [10] where the transformer is utilized in order to aggregate contextual information from other people and objects in the surrounding video.”
Perceiver: General Perception with Iterative Attention (2021, background): “Nevertheless, self-attention has been rapidly percolating into perception, for example as pieces of otherwise convolutional models for images (Bello et al., 2019; Cordonnier et al., 2020; Srinivas et al., 2021) and videos (Wang et al., 2018; Girdhar et al., 2019).”
FLAVA: A Foundational Language And Vision Alignment Model (2021, background): “The self-supervised pretraining paradigm has significantly advanced the state of the art across various domains, from natural language processing [6, 17–19, 23, 24, 28, 30, 61, 68, 73, 83–85], to computer vision [2, 5, 8, 12, 31, 33, 37, 59, 75, 104, 117], to speech recognition [4, 22, 42, 67, 119] ”
Visual Prompt Tuning (2022, background): “The triumph of the Transformer architecture also extends to various computer vision tasks, including image classification [16,48], object detection [7,45], semantic segmentation [66,83], panoptic segmentation [73], video understanding [21] and fewshot learning [15], surpassing previous state-of-the-”
Online Continual Learning with Contrastive Vision Transformer (2022, methodology): “Transformer model is firstly applied into machine translation tasks [74], and then, Transformers have become the state-of-the-art models for most natural language processing tasks [20,58,24,51,25].”
Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, methodology): “Trans-former models are now widely applied into visual question answering [87], object detection [88], segmentation [89], video understanding [90], etc.”
Transformers in Vision: A Survey (2021, methodology): “It is important to note that several of these works [69, 116, 185, 256] still employ (pretrained) CNNs to encode image/frame-level features in the videos on top of which Transformers are applied to model wide context.”
Is Space-Time Attention All You Need for Video Understanding? (2021, methodology): “However, we note that Trans-formers have been adopted on top of convolutional feature maps for action localization and recognition (Girdhar et al., 2019), video classiﬁcation (Wang et al., 2018b; Chen et al., 2018b), and group activity recognition (Gavrilyuk et al., 2020).”
Multiscale Vision Transformers (2021, background): “Incorporating downsampling, shift invariance, and shared weights, ConvNets are de-facto standard backbones for computer vision tasks for image [65, 62, 88, 90, 46, 12, 15, 34, 93, 81, 41] and video [87, 31, 11, 79, 69, 106, 96, 30, 105, 35, 29, 117, 57].”
ViViT: A Video Vision Transformer (2021, background): “Recently, these models were augmented by incorporating self-attention into their later layers to better capture long-range dependencies [75, 23, 79, 1].”
Mind the Gap: Assessing Temporal Generalization in Neural Language Models (2021, background): “This means that the ability of Transformers to condition on long-range context is already a useful feature for temporal generalization, even when we are not explicitly updating the model parameters with new data.”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, background): “They are also increasingly adopted in other areas, including vision (Carion et al., 2020; Dosovitskiy et al., 2020; Chen et al., 2020; Girdhar et al., 2019) and audio (Dong et al., 2018; Child et al., 2019).”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, background): “To assist more complicated action understanding, another growing set of research explores structural information in videos including temporal ordering [51, 88], object localization [4, 25, 32, 53, 74, 76], and implicit interactions between objects [4, 53].”
Omnivore: A Single Model for Many Visual Modalities (2022, methodology): “The Transformer architecture [92] originally proposed for NLP tasks has been successfully applied in computer vision on images [11, 21, 70, 88, 93, 94], video [5, 8, 28, 29, 52, 66], and 3D data [60, 68, 103].”
Predicting the Future from First Person (Egocentric) Vision: A Survey (2021, background): “Action transformer models for recognising and localizing human actions in video clips from third-person view have been introduced in [149].”
DistInit: Learning Video Representations Without a Single Labeled Video (2019, background): “Since then, numerous deep architectures [8, 9, 15, 16, 44, 51] have been proposed.”
Continual Learning with Lifelong Vision Transformer (2022, background): “Transformer is firstly proposed in [74] for machine translation tasks, and since then, transformer architectures have become the state-of-the-art models for natural language processing (NLP) tasks [19, 24, 48, 56].”
TransReID: Transformer-based Object Re-Identification (2021, methodology): “[8] use a variant of transformer architecture to aggregate contextual cues in a video relevant to a particular person.”