Audiovisual SlowFast Networks for Video Recognition

By Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, Christoph Feichtenhofer. 2020

256 citations recorded by the source corpus.

We present Audiovisual SlowFast Networks, an architecture for integrated audiovisual perception. AVSlowFast has Slow and Fast visual pathways that are deeply integrated with a Faster Audio pathway to model vision and sound in a unified representation. We fuse audio and visual features at multiple layers, enabling audio to contribute to the formation of hierarchical audiovisual concepts. To overcome training difficulties that arise from different learning dynamics for audio and visual modalities, we introduce DropPathway, which randomly drops the Audio pathway during training as an effective regularization technique. Inspired by prior studies in neuroscience, we perform hierarchical audiovisual synchronization to learn joint audiovisual features. We report state-of-the-art results on six video action classification and detection datasets, perform detailed ablation studies, and show the generalization of AVSlowFast to learn self-supervised audiovisual features. Code will be made available at: this https URL.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Perceiver: General Perception with Iterative Attention (2021, methodology): “Currently, separate feature extractors are used for each modality (Kaiser et al., 2017; Arandjelovic & Zisserman, 2018; Wang et al., 2020b; Chen et al., 2020b; Alayrac et al., 2020; Lee et al., 2020; Xiao et al., 2020) – it is not sensible to concatenate an audio spectrogram or a raw audio sequence ”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Relation to existing tasks The past few years have seen audio studied in computer vision tasks [245] for action classification [110,226], object categorization [125,234], source localization and tracking [14, 197, 212] and embodied navigation [33].”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “Nevertheless, the audio modality can serve as complementary information for more reliable and efficient HAR, and thus most of the audio-based HAR methods [25], [264], [265], [266] focused on taking advantage of multimodal deep learning techniques, which are discussed in”
Attention Bottlenecks for Multimodal Fusion (2021, methodology): “We note that on the entire Kinetics test set, our fusion model outperforms the visual only baseline by about 1% in top 1 accuracy (in line with other works [64] that demonstrate that audio for the large part does not improve performance for most Kinetics classes).”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, methodology): “As the next best method, AudioVisual SlowFast networks [64], reaches close performance only by applying a more advanced architecture and combining two losses.”
Balanced Multimodal Learning via On-the-fly Gradient Modulation (2022, methodology): “To demonstrate the advantage of OGM-GE, we make comparisons with two main-stream modulation approaches: Modality-Dropout [43], and Gradient-Blending [39].”
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (2021, methodology): “AVSlowFast [71] also utilizes audio, but is outperformed by our model with the same backbone (Ego-Exo*R101) on the S1 test set.”
Revisiting Skeleton-based Action Recognition (2021, background): “Existing studies have explored various modalities for feature representation, such as RGB frames [36, 31, 3], optical flows [28], audio waves [37], and human skeletons [39].”
Temporal Aggregate Representations for Long-Range Video Understanding (2020, methodology): “Selecting representative features can be as simple as sub-sampling the frames [9, 43], or pooling [40], to more complex RNNs [44].”
HMS: Hierarchical Modality Selection for Efficient Video Recognition (2021, unknown): “to improve video recognition performance [15, 45, 47].”