Stochasticity and Skip Connection Improve Knowledge Transfer

By Luong Trung Nguyen, Kwangjin Lee, Byonghyo Shim. 2021

12 citations recorded by the source corpus.

Deep neural networks have achieved state-of-the-art performance in various fields. However, DNNs need to be scaled down to fit real-word applications where memory and computation resources are limited. As a means to compress the network yet still maintain the performance of the network, knowledge distillation has brought a lot of attention. This technique is based on the idea to train a student network using the provided output of a teacher network. Deploying multiple teacher networks facilitates learning of the student network, however, it causes to some extent waste of resources. In the proposed approach, we generate multiple teacher networks from a teacher network by exploiting stochastic block and skip connection. Thus, they can play the role of multiple teacher networks and provide sufficient knowledge to the student network without additional resources. We observe the improved performance of student network with the proposed approach using several dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Lin Wang
  2. Kuk-Jin Yoon

In Computer Vision and Pattern Recognition

  1. Lin Wang
  2. Kuk-Jin Yoon

Citations in the corpus