Knowledge As Priors: Cross-Modal Knowledge Generalization for Datasets Without Superior Knowledge

By Long Zhao, Xi Peng, Yuxiao Chen, Mubbasir Kapadia, Dimitris N. Metaxas. 2020

73 citations recorded by the source corpus.

Cross-modal knowledge distillation deals with transferring knowledge from a model trained with superior modalities (Teacher) to another model trained with weak modalities (Student). Existing approaches require paired training examples exist in both modalities. However, accessing the data from superior modalities may not always be feasible. For example, in the case of 3D hand pose estimation, depth maps, point clouds, or stereo images usually capture better hand structures than RGB images, but most of them are expensive to be collected. In this paper, we propose a novel scheme to train the Student in a Target dataset where the Teacher is unavailable. Our key idea is to generalize the distilled cross-modal knowledge learned from a Source dataset, which contains paired examples from both modalities, to the Target dataset by modeling knowledge as priors on parameters of the Student. We name our method "Cross-Modal Knowledge Generalization" and demonstrate that our scheme results in competitive performance for 3D hand pose estimation on standard benchmark datasets.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Shijian Lu
3. Long Zhao
4. Jianping Gou
5. Xi Peng
6. S. Maybank
7. Sergey Tulyakov
8. Jian Ren

In Human-Computer Interaction

1. Dacheng Tao
2. Xi Peng
3. S. Maybank
4. Shijian Lu
5. Long Zhao

Citations in the corpus

• Knowledge Distillation: A Survey (2020, background): “The data or labels for some modalities might not be available during training or testing (Gupta et al., 2016; Garcia et al., 2018; Zhao et al., 2018; Roheda et al., 2018; Zhao et al., 2020).”
• SMIL: Multimodal Learning with Severely Missing Modality (2021, methodology): “Other works adapted MAML for domain generalization (Li et al. 2018; Qiao, Zhao, and Peng 2020) and knowledge distillation (Zhao et al. 2020).”
• FSDR: Frequency Space Domain Randomization for Domain Generalization (2021, methodology): “Most existing DG methods can be broadly categorized into multi-source DG [44, 13, 12, 34, 16, 33, 43, 42, 59, 36, 2, 10, 35] and single-source DG [64, 70, 77, 50] both of which strive to learn domainagnostic features by either learning a domain-invariant feature space [44, 13, 12, 16, 34, 59, 36, 10”
• 2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds (2022, methodology): “With the development of multi-modal computer vision, recent research apply knowledge distillation to transfer priors across different modalities, e.g. , exploiting extra 2D images in the training phase and improving the performance in the inference [51–55].”
• Learning to Learn Single Domain Generalization (2020, methodology): “Most previous methods either tried to learn a domain-invariant space to align domains [32, 9, 12, 21, 59] or aggregate domain-specific modules [29, 28].”