meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting

By Xu Sun, Xuancheng Ren, Shuming Ma, Houfeng Wang. 2017

171 citations recorded by the source corpus.

We propose a simple yet effective technique for neural network learning. The forward propagation is computed as usual. In back propagation, only a small subset of the full gradient is computed to update the model parameters. The gradient vectors are sparsified in such a way that only the top-$k$ elements (in terms of magnitude) are kept. As a result, only $k$ rows or columns (depending on the layout) of the weight matrix are modified, leading to a linear reduction ($k$ divided by the vector dimension) in the computational cost. Surprisingly, experimental results demonstrate that we can update only 1-4% of the weights at each back propagation pass. This does not result in a larger number of training iterations. More interestingly, the accuracy of the resulting models is actually improved rather than degraded, and a detailed analysis is given. The code is available at this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, background): “Gradient vector sparsification is another approach analyzed in [4, 40, 39, 5, 30, 34, 9, 3, 36, 21, 32].”
The Convergence of Sparsified Gradient Methods (2018, background): “[24] independently considered similar algorithms, but for privacy and regularization purposes, respectively.”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, background): “, for A [237] or for E [202], [233]), according to the neuronal selectivity [238] or even based on random [239]/probabilistic [240] selection.”
DAWNBench : An End-to-End Deep Learning Benchmark and Competition (2017, background): “Using 8-bit precision gives a 3x speed up on MNIST [10].”
M-FAC: Efficient Matrix-Free Approximations of Second-Order Information (2021, methodology): “The dynamic algorithm could parallelize gradient storage, or perform gradient compression [38].”
Fine-Tuning Language Models with Just Forward Passes (2023, methodology): “Several algorithms have been proposed to efficiently approximate backpropagation by sparsifying gradients [87, 96], approximating Jacobians [1, 18], and subsampling the computational graph [68, 2].”
Structured Pruning Adapters (2022, methodology): “A group of fine-pruning methods [21, 23, 30, 40, 42, 45] have combined pruning with fine-tuning to reduce model size while learning parameters for the target task.”