Fine-Pruning: Joint Fine-Tuning and Compression of a Convolutional Network with Bayesian Optimization

By Frederick Tung, Srikanth Muralidharan, Greg Mori. 2017

37 citations recorded by the source corpus.

When approaching a novel visual recognition problem in a specialized image domain, a common strategy is to start with a pre-trained deep neural network and fine-tune it to the specialized domain. If the target domain covers a smaller visual space than the source domain used for pre-training (e.g. ImageNet), the fine-tuned network is likely to be over-parameterized. However, applying network pruning as a post-processing step to reduce the memory requirements has drawbacks: fine-tuning and pruning are performed independently; pruning parameters are set once and cannot adapt over time; and the highly parameterized nature of state-of-the-art pruning methods make it prohibitive to manually search the pruning parameter space for deep networks, leading to coarse approximations. We propose a principled method for jointly fine-tuning and compressing a pre-trained convolutional network that overcomes these limitations. Experiments on two specialized image domains (remote sensing images and describable textures) demonstrate the validity of the proposed approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks (2018, methodology): “We note that the term fine-pruning has been used before in the context of transfer learning [42].”
A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers (2018, methodology): “These model compression techniques include weight pruning [10,11,5,31,25,9,29,21], sparsity regularization [30,19,32], weight clustering [10,3,26], and low rank approximation [6,7], etc.”
Constraint-Aware Deep Neural Network Compression (2018, background): “Why is cooling valuable? The cooling schedule induces a series of easier compression problems and allows the compression to adapt to the network structure as it changes over time [16].”
CLIP-Q: Deep Network Compression Learning by In-parallel Pruning-Quantization (2018, background): “Our focus in this paper is on deep network compression, which has the goal of making deep networks more compact [13, 16, 49].”