Listen to Look: Action Recognition by Previewing Audio

By Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, Lorenzo Torresani. 2019

295 citations recorded by the source corpus.

In the face of the video data deluge, today's expensive clip-level classifiers are increasingly impractical. We propose a framework for efficient action recognition in untrimmed video that uses audio as a preview mechanism to eliminate both short-term and long-term visual redundancies. First, we devise an ImgAud2Vid framework that hallucinates clip-level features by distilling from lighter modalities---a single frame and its accompanying audio---reducing short-term temporal redundancy for efficient clip-level recognition. Second, building on ImgAud2Vid, we further propose ImgAud-Skimming, an attention-based long short-term memory network that iteratively selects useful moments in untrimmed videos, reducing long-term temporal redundancy for efficient video-level recognition. Extensive experiments on four action recognition datasets demonstrate that our method achieves the state-of-the-art in terms of both recognition accuracy and speed.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, neutral): “Sa & Te Classification (action recognition) [61], [177], [181], [189], [190], [191], [192], [196], [232], Semantic segmentation [233] Video face recognition [22], [188], Action detection [179], [180], Action spotting [178], [187]”
Large Scale Audiovisual Learning of Sounds with Weakly Labeled Data (2020, methodology): “Further, training and inference in multi-modal models is computationally cumbersome, particularly for large datasets [Gao et al., 2019].”
Cross-Domain First Person Audio-Visual Action Recognition through Relative Norm Alignment (2021, methodology): “Audio has also been used as a preview for video skimming, due to its lightweight characteristics [20].”
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency (2021, background): “Apart from source separation, recent inspiring work integrates both audio and visual cues on an array of other tasks including self-supervised representation learning [57, 6, 8, 55, 41, 23], localizing sounds in video frames [7, 66, 71, 36], generating sounds from video [56, 84, 25, 47, 83], and act”
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition (2021, methodology): “Mini-Sports1M (assembled by [17]) is a subset of full Sports1M dataset [25] containing 30 videos per class in training and 10 videos per”
AR-Net: Adaptive Frame Resolution for Efficient Action Recognition (2020, unknown): “We quote the reported results on MultiAgent [59], AdaFrame[62] and ListenToLook[20] (here “(IA | R)” and “(MN | R)” are short for “(Image-Audio | ResNet-101)” and “(MobileNetV2 | ResNet-101)” mentioned in [20]).”
HMS: Hierarchical Modality Selection for Efficient Video Recognition (2021, methodology): “In addition, our approach is well-suited for both online and offline predictions where [7, 15] can only be used for offline recognition.”
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (2023, unknown): “Given a stream of audio, IMU, and RGB video data, a model must identify the keystep being performed at each frame, as well as decide which sensor(s) to use for subsequent time-steps.”
Dynamic Network Quantization for Efficient Video Inference (2021, unknown): “Moreover, unlike existing works, our framework requires neither complex RL policy gradients [60, 57, 63] nor additional modalities such as audio [14, 30] to learn dynamic policies.”
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (2021, background): “In videos, KD can incorporate information from alternate modalities like audio [24, 6], depth and flow [28, 64], or a combination [45], and object-level information [52].”
Adaptive Focus for Efficient Video Recognition (2021, methodology): “To address this issue, a number of recent works propose to reduce the inherent temporal redundancy in video recognition [26, 29, 45, 42, 16, 44, 25].”
2D or not 2D? Adaptive 3D Convolution Selection for Efficient Video Recognition (2020, methodology): “Compressed video [20] and audio [12, 20] are also utilized for further improvement in clip selection.”
VA-RED2: Video Adaptive Redundancy Reduction (2021, methodology): “Different from adaptive data sampling (Yeung et al., 2016; Wu et al., 2019; Korbar et al., 2019; Gao et al., 2019), in this paper, our goal is to remove feature map redundancy by deciding how many features need to be computed for temporal and channel dimensions per input basis, for efficient video…”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “Nevertheless, the audio modality can serve as complementary information for more reliable and efficient HAR, and thus most of the audio-based HAR methods [25], [264], [265], [266] focused on taking advantage of multimodal deep learning techniques, which are discussed in”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, methodology): “Futhermore, multi-modal learning has been used for several applications such as lip reading [4, 20, 21], audio-visual source separation and localization [3, 7, 28, 29, 39, 77, 87, 91, 113, 114], speech recognition [2, 73, 84], efficient inference [30, 56], egocentric action recognition [51] and navi”
FrameExit: Conditional Early Exiting for Efficient Video Recognition (2021, methodology): “This is commonly done by training agents to find which frame to observe next [55, 52, 57, 8], ranking frames based on saliency [29], gating frames [25], skipping RNN states [4], or by using audio to select relevant frames [15, 29].”
Balanced Multimodal Learning via On-the-fly Gradient Modulation (2022, background): “, action recognition [10, 23, 29], audiovisual speech recognition [16, 33], and visual question answering [2, 19].”
AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition (2021, methodology): “, 2019)) or selecting salient frames/clips conditioned on the input (Yeung et al., 2016; Wu et al., 2019b; Korbar et al., 2019; Gao et al., 2020).”