Faster gaze prediction with dense networks and Fisher pruning

By Lucas Theis, Iryna Korshunova, Alykhan Tejani, Ferenc Huszár. 2018

231 citations recorded by the source corpus.

Predicting human fixations from images has recently seen large improvements by leveraging deep representations which were pretrained for object recognition. However, as we show in this paper, these networks are highly overparameterized for the task of fixation prediction. We first present a simple yet principled greedy pruning method which we call Fisher pruning. Through a combination of knowledge distillation and Fisher pruning, we obtain much more runtime-efficient architectures for saliency prediction, achieving a 10x speedup for the same AUC performance as a state of the art network on the CAT2000 dataset. Speeding up single-image gaze prediction is important for many real-world applications, but it is also a crucial step in the development of video saliency models, where the amount of data to be processed is substantially larger.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

The State of Sparsity in Deep Neural Networks (2019, methodology): “…and this property has been leveraged to signiﬁcantly reduce the cost associated with the deployment of deep neural networks, and to enable the deployment of state-of-the-art models in severely resource constrained environments (Theis et al., 2018; Kalchbrenner et al., 2018; Valin & Skoglund, 2018).”
Training Neural Networks with Fixed Sparse Masks (2021, methodology): “Most relevant to our work, Theis et al. [46] propose utilizing the Fisher to prune and decrease the overall number of parameters for gaze prediction, and Liu et al. [30] also use the Fisher to discover groups of parameters to prune from common backbone architectures.”
Wireless Network Intelligence at the Edge (2018, methodology): “, based on the Fisher information of the weights [117].”
Efficient Structured Pruning and Architecture Searching for Group Convolution (2018, background): “, [39, 29]) to improve the quality of explored models, tuning the pruning hyperparameters to achieve higher model accuracy after the fine-tuning phase, and investigating the contribution of the regularisation effect on the better accuracy from CondenseNet.”
Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks (2018, methodology): “…et al., 2017), (cid:96) 2 of ﬁlter weights and ﬁlter weights of the next layer (Jiang et al., 2018), ﬁrst-order Taylor expansion on the loss (Molchanov et al., 2017; Lin et al., 2018), Fisher information of the loss (Theis et al., 2018), and the variance of the max activation (Yoon et al., 2018).”
Movement Pruning: Adaptive Sparsity by Fine-Tuning (2020, background): “Theis et al. [2018] and Ding et al. [2019] use the absolute value or the square value of the gradient.”
Artificial neural networks condensation: A strategy to facilitate adaption of machine learning in medical settings by reducing computational burden (2018, methodology): “There is another method called Fisher Pruning [8], where they assume the trained model is located at a local minimum of its loss; thus by estimating the Hessian matrix they would be able to know which direction gets the smallest estimated increase in loss and prune the weight that has the maximum…”
WoodFisher: Efficient second-order approximations for model compression (2020, methodology): “, 2010), as well for model compression by the work called Fisher-pruning (Theis et al., 2018).”
Comparing Rewinding and Fine-tuning in Neural Network Pruning (2020, methodology): “, 2017) and those that prune based on other information (e.g., Le Cun et al., 1990; Theis et al., 2018; Lee et al., 2019).”
M-FAC: Efficient Matrix-Free Approximations of Second-Order Information (2021, background): “Singh and Alistarh [37] showed that it can be scaled to DNNs by blockwise approximation, and that it provides better approximations of the loss than K-FAC-based [44], or diagonal [39] approximations [7], leading to more accurate pruning.”
WoodFisher: Efficient Second-Order Approximation for Neural Network Compression (2020, methodology): “This approximation is therefore fairly common [15, 20, 31], although there is relatively little work examining the quality of this approximation in the context of neural networks.”
Fast Sparse ConvNets (2019, unknown): “Due to the difficultly of accelerating sparse computation, channel pruning approaches have been preferred [10, 5, 29, 28, 42, 13].”
A comprehensive survey on model compression and acceleration (2020, methodology): “In addition, Crowley et al. (2018) applied L 1-norm (Li et al. 2017) and Fisher (Theis et al. 2018) pruning on ResNets (He et al. 2016a) and DenseNets (Huang et al. 2017) architectures and found that for a given parameter budget, reduced networks (smaller version of the original network) trained fro”
Differentiable Joint Pruning and Quantization for Hardware Efficiency (2020, background): “To explore inter-channel dependency, many works explore second-order Taylor expansion, such as [26][24].”
A review of convolutional neural network architectures and their optimizations (2022, methodology): “developed an optimization method that combines Fisher pruning and knowledge transfer (Theis et al. 2018).”
BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction (2021, methodology): “A similar method for pruning the pre-activation has been proposed in Theis et al. (2018).”