Anticipative Video Transformer

By Rohit Girdhar, Kristen Grauman. 2021

285 citations recorded by the source corpus.

We propose Anticipative Video Transformer (AVT), an end-to-end attention-based video modeling architecture that attends to the previously observed video in order to anticipate future actions. We train the model jointly to predict the next action in a video sequence, while also learning frame feature encoders that are predictive of successive future frames’ features. Compared to existing temporal aggregation strategies, AVT has the advantage of both maintaining the sequential progression of observed actions while still capturing long-range dependencies—both critical for the anticipation task. Through extensive experiments, we show that AVT obtains the best reported performance on four popular action anticipation benchmarks: EpicKitchens-55, EpicKitchens-100, EGTEA Gaze+, and 50-Salads; and it wins first place in the EpicKitchens-100 CVPR’21 challenge.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “[45] Rohit Girdhar and Kristen Grauman.”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “…video offers a host of interesting challenges, such as human-object interactions [26,46,163], activity recognition [110,139,243], anticipation [4,75,86,144,205], video summarization [48, 129,131,147,148,232], detecting hands [16,134], parsing social interactions [66,168,231], and inferring the…”
Omnivore: A Single Model for Many Visual Modalities (2022, methodology): “The Transformer architecture [92] originally proposed for NLP tasks has been successfully applied in computer vision on images [11, 21, 70, 88, 93, 94], video [5, 8, 28, 29, 52, 66], and 3D data [60, 68, 103].”
Revisiting Feature Prediction for Learning Visual Representations from Video (2024, methodology): “…the video encoder and the predictor network simultaneously, while preventing collapse by using a supervised action forecasting loss (Girdhar and Grauman, 2021), or by using the representations of distant clips as negative samples in a contrastive loss (Han et al., 2019, 2020; Tan et al.,…”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “Similarly, in action anticipation and human motion forecasting, high-level semantic or low level actions are predicted using visual history [1, 11,19,22,31–33,37,40,46–48,55,58,72,75,100,119,120].”
Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos (2022, methodology): “What space should the model predict on? Recent approaches [25,26,28,72] have been proposed to predict the discrete future action category given a sequence of frames as inputs, namely action anticipation.”
Expanding Language-Image Pretrained Models for General Video Recognition (2022, methodology): “1) In [33,3,18], each frame is encoded separately, resulting in no temporal interaction before final aggregation.”