Modality Distillation with Multiple Stream Networks for Action Recognition

By Nuno C. Garcia, Pietro Morerio, Vittorio Murino. 2018

214 citations recorded by the source corpus.

Diverse input data modalities can provide complementary cues for several tasks, usually leading to more robust algorithms and better performance. However, while a (training) dataset could be accurately designed to include a variety of sensory inputs, it is often the case that not all modalities are available in real life (testing) scenarios, where a model has to be deployed. This raises the challenge of how to learn robust representations leveraging multimodal data in the training stage, while considering limitations at test time, such as noisy or missing modalities. This paper presents a new approach for multimodal video action recognition, developed within the unified frameworks of distillation and privileged information, named generalized distillation. Particularly, we consider the case of learning representations from depth and RGB videos, while relying on RGB data only at test time. We propose a new approach to train an hallucination network that learns to distill depth features through multiplicative connections of spatiotemporal representations, leveraging soft labels and hard labels, as well as distance between feature maps. We report state-of-the-art results on video action classification on the largest multimodal dataset available for this task, the NTU RGB+D, as well as on the UWA3DII and Northwestern-UCLA.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “Efﬁcient and effective action recognition under its cross-modal task scenarios can be successfully realized (Thoker and Gall 2019; Luo et al. 2018; Garcia et al. 2018; Hao and Zhang 2019; Wu et al. 2019b; Zhang et al. 2020c).”
Cross-View Policy Learning for Street Navigation (2019, unknown): “by adding L2 losses for reconstructing all layers of the depth network and a cross entropy distillation loss for a missing network [12].”
Deep Multimodal Fusion by Channel Exchanging (2020, background): “Joining the success of deep learning, multimodal fusion is recently specified as deep multimodal fusion by introducing end-to-end neural integration of multiple modalities [37], and it has exhibited remarkable benefits against the unimodal paradigm in semantic segmentation [28, 44], action recogniti”
Audio-Visual Model Distillation Using Acoustic Images (2019, methodology): “Other works such as [11, 20] utilize RGB video images and depth information to learn feature representations through modality hallucination.”
Source-Free Domain Adaptation for Semantic Segmentation (2021, background): “Since then, a variety of KD methods has been presented for model compression [28, 2, 50, 32], domain adaptation [52, 53], and multi-modal learning [51, 14, 10].”
Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition (2020, background): “The existing video analysis algorithms usually require for large-scale data transmission, several works focus on how to use relative fewer input frames for action recognition while maintaining competitive accuracy (Luo et al. 2018; Garcia, Morerio, and Murino 2018).”
Cross-Modal Knowledge Distillation for Action Recognition (2019, background): “Similarly, [19] proposed a multi-modal action recognition framework that uses multiple data modalities at training time.”
MARS: Motion-Augmented RGB Stream for Action Recognition (2019, background): “[9] developed a distillation framework for action recognition with their four-step process that hallu-cinates depth features from RGB frames.”