Cross Modal Distillation for Supervision Transfer

By Saurabh Gupta, Judy Hoffman, Jitendra Malik. 2015

618 citations recorded by the source corpus.

In this work we propose a technique that transfers supervision between images from different modalities. We use learned representations from a large labeled modality as supervisory signal for training representations for a new unlabeled paired modality. Our method enables learning of rich representations for unlabeled modalities and can be used as a pre-training procedure for new modalities with limited labeled data. We transfer supervision from labeled RGB images to unlabeled depth and optical flow images and demonstrate large improvements for both these cross modal supervision transfers.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Growing a Brain: Fine-Tuning by Increasing Model Capacity (2017, background): “Transfer learning is related to both multi-task learning [32, 4, 28, 10, 11, 45, 24, 2] and learning novel categories from few examples [49, 19, 21, 35, 51, 22, 5, 13, 50, 47, 31].”
Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction (2016, methodology): “The method ﬁrst trains individual detectors on the RGB and D domains, using the Fast R-CNN [14].”
Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning (2017, neutral): “To do this, they used a form of cross-modal distillation (Gupta et al 2016), and trained an audio CNN to predict which semantic labels a pre-trained visual CNN will assign to a paired audio-visual example.”
Stacked Generative Adversarial Networks (2016, methodology): “[51, 20] use the intermediate representations of one pre-trained DNN to guide another DNN in the context of knowledge transfer.”
Generative Image Modeling Using Style and Structure Adversarial Networks (2016, methodology): “For networks pre-trained on only RGB data, we fine-tune them on both the RGB and surface normal inputs separately and average the detection results during testing as [66].”
SoundNet: Learning Sound Representations from Unlabeled Video (2016, methodology): “Our method builds upon teacher-student models [3, 9] and dark knowledge transfer [13].”
NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “Learning representations from a large labeled modality for transfer learning for the smaller-scale new modalities has attracted a lot of research attention and has been applied to different tasks recently [112].”
Contrastive Representation Distillation (2019, background): “, 2016) or to depth (Gupta et al., 2016) processing network, such that deep features for an image and the associated sound or depth features are highly correlated.”
Unsupervised Domain Adaptation with Residual Transfer Networks (2016, background): “Note that the state of the art deep feature adaptation methods [5, 6, 7] generally assume classifiers can be shared on adapted deep features.”
Knowledge Distillation: A Survey (2020, methodology): “rivacy and security of data (Vongkulbhisal et al., 2019; Papernot et al., 2017). 6.3 Cross-Modal Distillation The data or labels for some modalities might not be available during training or testing (Gupta et al., 2016; Garcia et al., 2018; Zhao et al., 2018; Roheda et al., 2018). For this reason it”
Incremental Learning of Object Detectors without Catastrophic Forgetting (2017, background): “It has found several applications in domain adaptation and model compression [17, 34, 39].”
Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images (2015, methodology): “For the VGG network [22], we use the weights from [12] without fine tuning.”
Cognitive Mapping and Planning for Visual Navigation (2017, methodology): “We transfer supervision from RGB images to Depth images using cross modal distillation (Gupta et al. 2016) between RGB-D image pairs rendered from meshes in the training set to obtain a pretrained ResNet-50 model to represent Depth images.”
Data Distillation: Towards Omni-Supervised Learning (2017, background): “Cross modal distillation [13] is proposed to address the problem of limited labels in a certain modality.”
Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization (2018, methodology): “Pre-trained visual classiﬁers (noisy predictions) were used as supervision for pre-training audio models [20] as well as CNN models with depth images as input [43].”
Cross and Learn: Cross-Modal Self-Supervision (2018, background): “s of utilizing multimodal data in order to solve a variety of tasks. For example [33,8] show that RGB and optical ﬂow are complementary modalities in the context of action recognition. S.Gupta et al. [11] train a feature representation for depth data by distilling the knowledge from a teacher networ”
On the Efficacy of Knowledge Distillation (2019, background): “In terms of applications of knowledge distillation, prior work has found knowledge distillation to be useful for sequence modeling [16, 4], semi-supervised learning [21], domain adaptation [19], multi-modal learning [9] and so on.”
Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation (2019, methodology): “, using the same methods, transferred the knowledge among data in different modals [10].”
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (2021, background): “In videos, KD can incorporate information from alternate modalities like audio [24, 6], depth and flow [28, 64], or a combination [45], and object-level information [52].”
Self-Supervised Learning by Cross-Modal Audio-Video Clustering (2019, background): “Several works [2, 17] train a fully-supervised encoder on one modality and distill its discriminative knowledge to an encoder of a different modality.”