In Teacher We Trust: Learning Compressed Models for Pedestrian Detection

By Jonathan Shen, Noranart Vesdapunt, Vishnu Naresh Boddeti, Kris M. Kitani. 2016

30 citations recorded by the source corpus.

Deep convolutional neural networks continue to advance the state-of-the-art in many domains as they grow bigger and more complex. It has been observed that many of the parameters of a large network are redundant, allowing for the possibility of learning a smaller network that mimics the outputs of the large network through a process called Knowledge Distillation. We show, however, that standard Knowledge Distillation is not effective for learning small models for the task of pedestrian detection. To improve this process, we introduce a higher-dimensional hint layer to increase information flow. We also estimate the variance in the outputs of the large network and propose a loss function to incorporate this uncertainty. Finally, we attempt to boost the complexity of the small network without increasing its size by using as input hand-designed features that have been demonstrated to be effective for pedestrian detection. We succeed in training a model that contains $400\times$ fewer parameters than the large network while outperforming AlexNet on the Caltech Pedestrian Dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, background): “, 2019), pedestrian detection (Shen et al., 2016), video classification (Bhardwaj et al.”
Learning Metrics From Teachers: Compact Networks for Image Embedding (2019, methodology): “Most literature on network distillation focuses on image classification, but recently several works have investigated applying the theory to object detection [3] and pedestrian detection [28].”
Training Student Networks for Acceleration with Conditional Adversarial Networks (2018, methodology): “Variants of knowledge distillation has also been applied to tasks in other domains [5, 23, 30, 33] A recent preprint [17] presented promising preliminary results on CIFAR-10 by learning a small ResNet from a large ResNet.”
Learning Efficient Object Detection Models with Knowledge Distillation (2017, background): “[36] consider the effect of distillation and hint frameworks in learning a compact object detection model.”
Knowledge Distillation for End-to-End Person Search (2019, unknown): “When moving from classiﬁcation to detection, as only few works [8, 30, 46] attempted, complex modelling questions arise as to where and how to supervise, which are not entirely answered yet.”
Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks (2017, methodology): “Variants of knowledge distillation has also been applied to many different tasks, such as semantic segmentation [31], pedestrian detection [35], face recognition [24], metric learning[6], reinforcement learning [38] and for regularization[34].”
A comprehensive survey on model compression and acceleration (2020, methodology): “…research, Chen et al. (2017) introduces KD based fast end-to-end trainable framework for multi-class object detection, which makes it different from Shen et al. (2016), where the proposed method was limited to single pedestrian detection only, making infeasible for multi-class object detection.”