Adapting Models to Signal Degradation using Distillation

By Jong-Chyi Su, Subhransu Maji. 2016

32 citations recorded by the source corpus.

Model compression and knowledge distillation have been successfully applied for cross-architecture and cross-domain transfer learning. However, a key requirement is that training examples are in correspondence across the domains. We show that in many scenarios of practical importance such aligned data can be synthetically generated using computer graphics pipelines allowing domain adaptation through distillation. We apply this technique to learn models for recognizing low-resolution images using labeled high-resolution images, non-localized objects using labeled localized objects, line-drawings using labeled color images, etc. Experiments on various fine-grained recognition datasets demonstrate that the technique improves recognition performance on the low-quality data and beats strong baselines for domain adaptation. Finally, we present insights into workings of the technique through visualizations and relating it to existing literature.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, background): “Besides, these cross-model distillation methods also transfer the knowledge amongmultiple domains (Kundu et al. 2019; Chen et al. 2019c; Su and Maji 2017).”
Learning Metrics From Teachers: Compact Networks for Image Embedding (2019, background): “The improvement by the distillation is more noticeable on the Cars-196 dataset which is also observed in [33].”
Low-Resolution Face Recognition in the Wild via Selective Knowledge Distillation (2018, background): “Su and Maji [51] proposed cross quality distillation to learn models for recognizing low-resolution images, non-localized objects and line-drawings by using labeled high-resolution images, labeled localized objects and color images, respectively.”
When Does Self-supervision Improve Few-shot Learning? (2019, background): “natural categories [56].”
Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation (2019, methodology): “On CIFAR-10S, we consider other transformations instead of the grayscale domain: (1) cropping the center of the image, (2,3) reducing the image resolution [44], followed by upsampling or (4) replacing with 32x32 ImageNet images of the same class [10].”
Efficient Low-Resolution Face Recognition via Bridge Distillation (2020, background): “Su and Maji [43] utilized cross quality distillation to learn models for recognizing low-resolution images.”