Data-free Knowledge Distillation for Object Detection

By Akshay Chawla, Hongxu Yin, Pavlo Molchanov, José Manuel Álvarez. 2021

88 citations recorded by the source corpus.

We present DeepInversion for Object Detection (DIODE) to enable data-free knowledge distillation for neural networks trained on the object detection task. From a data-free perspective, DIODE synthesizes images given only an off-the-shelf pre-trained detection network and without any prior domain knowledge, generator network, or pre-computed activations. DIODE relies on two key components—first, an extensive set of differentiable augmentations to improve image fidelity and distillation effectiveness. Second, a novel automated bounding box and category sampling scheme for image synthesis enabling generating a large number of images with a diverse set of spatial and category objects. The resulting images enable data-free knowledge distillation from a teacher to a student detector, initialized from scratch.In an extensive set of experiments, we demonstrate that DIODE’s ability to match the original training distribution consistently enables more effective knowledge distillation than out-of-distribution proxy datasets, which unavoidably occur in a data-free setup given the absence of the original domain knowledge.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation (2022, background): “By synthesize categoryconditioned objects through inverse mapping, [3] proposed a data-free distillation technology applicable for object detection, but the method would trigger dream-image.”
Memory Efficient Continual Learning with Transformers (2022, methodology): “[9] Akshay Chawla, Hongxu Yin, Pavlo Molchanov, and Jose Alvarez.”
Knowledge Distillation: A Survey (2020, background): “…have been proposed to overcome problems with unavailable data arising from privacy, legality, security and conﬁdentiality concerns (Chen et al. 2019a; Lopes et al. 2017; Nayak et al. 2019; Micaelli and Storkey 2019; Haroush et al. 2020; Ye et al. 2020; Nayak et al. 2021; Chawla et al. 2021).”
Memory Efficient Continual Learning for Neural Text Classification (2022, methodology): “While several different data sources can be used to populate U , such as using auxiliary external data Zhang et al. (2020) or generating synthetic data Chawla et al. (2021), in this work we populate the buffer using covariates from previous tasks selected with Reservoir Sampling (Vitter, 1985).”