Quantization Mimic: Towards Very Tiny CNN for Object Detection

By Yi Wei, Xinyu Pan, Hongwei Qin, Junjie Yan. 2018

101 citations recorded by the source corpus.

In this paper, we propose a simple and general framework for training very tiny CNNs (e.g. VGG with the number of channels reduced to \(\frac{1}{32}\)) for object detection. Due to limited representation ability, it is challenging to train very tiny networks for complicated tasks like detection. To the best of our knowledge, our method, called Quantization Mimic, is the first one focusing on very tiny networks. We utilize two types of acceleration methods: mimic and quantization. Mimic improves the performance of a student network by transfering knowledge from a teacher network. Quantization converts a full-precision network to a quantized one without large degradation of performance. If the teacher network is quantized, the search scope of the student network will be smaller. Using this feature of the quantization, we propose Quantization Mimic. It first quantizes the large network, then mimic a quantized small network. The quantization operation can help student network to better match the feature maps from teacher network. To evaluate our approach, we carry out experiments on various popular CNNs including VGG and Resnet, as well as different detection frameworks including Faster R-CNN and R-FCN. Experiments on Pascal VOC and WIDER FACE verify that our Quantization Mimic algorithm can be applied on various settings and outperforms state-of-the-art model acceleration methods given limited computing resouces.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “For example, quantized knowledge distillation, which can be seen as a parameter pruning method, integrates network quantization into the teacher-student architectures (Polino et al., 2018; Mishra and Marr, 2018; Wei et al., 2018).”
Deep Learning for Generic Object Detection: A Survey (2018, methodology): “In order to remove the redundancy in the model, there has been growing research interest in designing compact and lightweight networks [29, 4, 115, 108, 165, 293], network compression and acceleration [42, 118, 248, 151, 154, 275].”
QueryDet: Cascaded Sparse Query for Accelerating High-Resolution Small Object Detection (2021, methodology): “Note that our approach is designed to save the computation spatially, so it is compatible with other accelerating methods like light-weighted backbones [49], model pruning [17], model quantization [56] and knowledge distillation [6].”
Deep Face Recognition: A Survey (2018, background): “Moreover, some wellknown compressed networks such as Pruning [72], [73], [131], BinaryNets [39], [92], [162], [38], Mimic Networks [116], [214], also have potential to be introduced into FR.”
Distilling Object Detectors via Decoupled Features (2021, background): “Techniques such as quantization [19, 58, 31, 57, 62], pruning [2, 17, 20], network design [55, 49, 15, 18] and knowledge distillation [56, 6] have been developed to overcome this dilemma and achieve an efficient inference on detection task.”
Fully Quantized Network for Object Detection (2019, background): “There have been several works on quantizing detectors to use 8 bits [17, 28].”
Data-free Knowledge Distillation for Object Detection (2021, background): “Unlike existing approaches for distillation of object detectors [19, 37, 42], our approach does not require access to images or labels from the teacher’s training data.”
Distilling Knowledge via Knowledge Review (2021, background): “It forms a very general line, applicable to almost all network architectures and can combine with many other strategies, such as network pruning and quantization [32], to further improve network design.”