Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks

By Jason Kuen, Xiangfei Kong, Zhe L. Lin, G. Wang, Jianxiong Yin, Simon See, Yap-Peng Tan. 2018

29 citations recorded by the source corpus.

It is desirable to train convolutional networks (CNNs) to run more efficiently during inference. In many cases however, the computational budget that the system has for inference cannot be known beforehand during training, or the inference budget is dependent on the changing real-time resource availability. Thus, it is inadequate to train just inference-efficient CNNs, whose inference costs are not adjustable and cannot adapt to varied inference budgets. We propose a novel approach for cost-adjustable inference in CNNs - Stochastic Downsampling Point (SDPoint). During training, SDPoint applies feature map downsampling to a random point in the layer hierarchy, with a random downsampling ratio. The different stochastic downsampling configurations known as SDPoint instances (of the same model) have computational costs different from each other, while being trained to minimize the same prediction loss. Sharing network parameters across different instances provides significant regularization boost. During inference, one may handpick a SDPoint instance that best fits the inference budget. The effectiveness of SDPoint, as both a cost-adjustable inference approach and a regularizer, is validated through extensive experiments on image classification.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Thomas S. Huang
2. Jiahui Yu
3. Zheng Zhang
4. Gao Huang
5. Alan L. Yuille
6. Zhenda Xie
7. Stephen Lin
8. Chenglong Bao

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Ming-Hsuan Yang
4. Xiaogang Wang
5. Alan L. Yuille
6. Trevor Darrell
7. Chunhua Shen
8. Yi Yang

Citations in the corpus

• Slimmable Neural Networks (2018, methodology): “017) introduced additional controller modules or gating functions to determine the computation graph based on the current input. Amthor et al. (2016), Veit & Belongie (2017), Huang et al. (2017), Kuen et al. (2018) and Hu et al. (2017) implanted earlyexiting prediction branches to reduce the ave”
• Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation (2019, background): “Another extension of the layer-wise dropout solution is to design early-exiting prediction branches to reduce the average execution depth in inference [17, 1, 35, 23].”
• Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolation (2020, methodology): “In [19], various predefined downsampling configurations are randomly used during training, and the appropriate configuration is applied according to the computation budget during inference.”
• InstaNAS: Instance-aware Neural Architecture Search (2018, methodology): “Several conditional computation methods have been proposed to dynamically execute different modules of a model on a per-example basis [2, 12, 14, 27, 32, 30].”