Long-Term Feature Banks for Detailed Video Understanding

By Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, Ross B. Girshick. 2018

526 citations recorded by the source corpus.

To understand the world, we humans constantly need to relate the present to the past, and put events in context. In this paper, we enable existing video models to do the same. We propose a long-term feature bank—supportive information extracted over the entire span of a video—to augment state-of-the-art video models that otherwise would only view short clips of 2-5 seconds. Our experiments demonstrate that augmenting 3D convolutional networks with a long-term feature bank yields state-of-the-art results on three challenging video datasets: AVA, EPIC-Kitchens, and Charades. Code is available online.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Multiview Transformers for Video Recognition (2022, methodology): “To model multiscale temporal information in videos, previous approaches such as SlowFast [23] have processed videos with two streams, using a “Fast” stream operating at high frame rates and a “Slow” stream at low frame rates, or employed graph neural networks to model long-range interactions [4, 77]”
Contrastive Bidirectional Transformer for Temporal Representation Learning (2019, methodology): “An alternative approach uses a long-term feature bank [34], which extracts information over the entire span of a video and selects relevant interactions by cross-”
SlowFast Networks for Video Recognition (2018, methodology): “The region proposals used in our submission are originating from the LFB project [25].”
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures (2019, neutral): “Method modality mAP 2-Strm. [20] (from [18]) RGB+Flow 18.6 Asyn-TF [18] RGB+Flow 22.4 CoViAR [28] Compressed 21.9 MultiScale TRN [33] RGB 25.2 I3D [3] RGB 32.9 I3D [3] (from [25]) RGB 35.5 I3D-NL [25] RGB 37.5 STRG [26] RGB 39.7 LFB [27] RGB 42.5 SlowFast [6] RGB+RGB 45.”
Multiscale Vision Transformers (2021, background): “Incorporating downsampling, shift invariance, and shared weights, ConvNets are de-facto standard backbones for computer vision tasks for image [65, 62, 88, 90, 46, 12, 15, 34, 93, 81, 41] and video [87, 31, 11, 79, 69, 106, 96, 30, 105, 35, 29, 117, 57].”
ViViT: A Video Vision Transformer (2021, background): “Recently, these models were augmented by incorporating self-attention into their later layers to better capture long-range dependencies [75, 23, 79, 1].”
Hierarchical Conditional Relation Networks for Video Question Answering (2020, background): “Multiscale temporal structure can be modeled by either mixing short and long term convolutional ﬁlters [37] or combining pre-extracted frame features non-local operators [32, 18].”
Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video (2020, neutral): “13 FB [77] 60.”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, methodology): “Prior arts use 3D volumn [90], object/humancentric [67, 91], or other forms [71, 92] as video representations.”
Audiovisual SlowFast Networks for Video Recognition (2020, methodology): “Comparing to LFB [81], that uses an object detector to localize objects, AVSlowFast achieve similar performance as for nouns (objects) on both the seen and unseen test sets, whereas SlowFast without audio is largely lacking behind LFB (-4.4% compared to LFB on val noun), which is intuitive as sound…”
Anticipative Video Transformer (2021, methodology): “The typical approach to solving long-term predictive reasoning tasks involves extracting frame or clip level features using standard architectures [12, 83, 88], followed by aggregation using clustering [32, 60], recurrence [23, 24, 41], or attention [28, 57, 74, 92] based models.”
Retrieval-Enhanced Machine Learning (2022, unknown): “Wu et al. [63] proposed using a long-term feature bank for detailed video understanding.”
Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition (2019, background): “Recent advances in this field focus on capturing longer-range temporal dependencies [55, 56, 63].”
LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities (2020, methodology): “Implementation Details: For the LFB model, we use a history window size of 10 and aggregate the features using max-pooling, as described in [62].”
Revisiting the “Video” in Video-Language Understanding (2022, background): “…analysis was tremendous: later datasets were designed to capture a richer distribution of temporal understanding [6,13,47] with better dis-entanglement of such cues [34], and model designs evolved further to better capture the now necessary dynamics to address these improved tasks [9–11,31,52].”
Reducto: On-Camera Filtering for Resource-Efficient Real-Time Video Analytics (2020, background): “The idea of filtering frames based on their features is widely seen in the CV community [18, 29, 44, 60, 70, 74, 75].”
Listen to Look: Action Recognition by Previewing Audio (2019, background): “Recent work includes capturing long-term temporal structure via recurrent networks [74, 11] or ranking functions [17], pooling across space and/or time [63, 23], modeling hierarchical or spatiotemporal information in videos [44, 58], building long-term temporal relations [67, 77], or boosting accura”
Ego-Topo: Environment Affordances From Egocentric Video (2020, background): “Architectures for composite activity learn to encode atomic action “primitives” that are aggregated over the full time extent of the video [17, 30, 31], memory-based models record a recurrent network’s state [53], and 3D convnets augmented with longterm feature banks provide temporal context [68].”