Graph Distillation for Action Detection with Privileged Modalities

By Zelun Luo, Jun-Ting Hsieh, Lu Jiang, Juan Carlos Niebles, Li Fei-Fei. 2017

118 citations recorded by the source corpus.

We propose a technique that tackles action detection in multimodal videos under a realistic and challenging condition in which only limited training data and partially observed modalities are available. Common methods in transfer learning do not take advantage of the extra modalities potentially available in the source domain. On the other hand, previous work on multimodal learning only focuses on a single domain or task and does not handle the modality discrepancy between training and testing. In this work, we propose a method termed graph distillation that incorporates rich privileged information from a large-scale multimodal dataset in the source domain, and improves the learning in the target domain where training data and modalities are scarce. We evaluate our approach on action classification and detection tasks in multimodal videos, and show that our model outperforms the state-of-the-art by a large margin on the NTU RGB+D and PKU-MMD benchmarks. The code is released at http://alan.vision/eccv18_graph/.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, background): “…mutual teacher–student networks (Thoker and Gall 2019), multiple stream networks (Garcia et al. 2018), spatiotemporal distilled dense-connectivity network (Hao and Zhang 2019), graph distillation (Luo et al. 2018) and multi-teacher to multi-student networks (Wu et al. 2019b; Zhang et al. 2020c).”
Small Sample Learning in Big Data Era (2018, background): “…recent years, the dual learning framework is successfully applied to visual question answering(VQA) (Li et al., 2018c), semantic image segmentation (Luo et al., 2017a), image-to-image translation (Zhu et al., 2017b; Yi et al., 2017), zero-shot visual recognition (Chen et al., 2018b), and neural…”
Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition (2020, background): “The existing video analysis algorithms usually require for large-scale data transmission, several works focus on how to use relative fewer input frames for action recognition while maintaining competitive accuracy (Luo et al. 2018; Garcia, Morerio, and Murino 2018).”
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (2021, background): “In videos, KD can incorporate information from alternate modalities like audio [24, 6], depth and flow [28, 64], or a combination [45], and object-level information [52].”
Cross-Modal Knowledge Distillation for Action Recognition (2019, unknown): “For instance, [18] use a graph-based distillation method for action recognition that is able to distill information from multiple modalities during training.”
Better and Faster: Knowledge Transfer from Multiple Self-supervised Learning Tasks via Graph Distillation for Video Classification (2018, background): “More recently, [ Luo et al. , 2017 ] considers multi-modal data as privileged information and extends GD paradigm to a graph-based form.”
MARS: Motion-Augmented RGB Stream for Action Recognition (2019, unknown): “Another recently proposed graph distillation approach [25] dynamically leverages information across different modalities.”
Cross-View Policy Learning for Street Navigation (2019, methodology): “[26] learned the direction of distillation between modalities, considering a cosine distillation loss and a representation loss.”