Learning Discrete Weights Using the Local Reparameterization Trick

By Oran Shayer, Dan Levi, Ethan Fetaya. 2017

92 citations recorded by the source corpus.

Recent breakthroughs in computer vision make use of large deep neural networks, utilizing the substantial speedup offered by GPUs. For applications running on limited hardware, however, high precision real-time processing can still be a challenge. One approach to solving this problem is training networks with binary or ternary weights, thus removing the need to calculate multiplications and significantly reducing memory size. In this work, we introduce LR-nets (Local reparameterization networks), a new method for training neural networks with discrete weights using stochastic parameters. We show how a simple modification to the local reparameterization trick, previously used to train Gaussian distributed weights, enables the training of discrete weights. Using the proposed training we test both binary and ternary models on MNIST, CIFAR-10 and ImageNet benchmarks and reach state-of-the-art results on most experiments.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Differentiable Model Compression via Pseudo Quantization Noise (2021, background): “Shayer, Levi, and Fetaya (2018) relies on a Central Limit Theorem (CLT) application, however this prevents weights from converging to a deterministic value, which would break the assumptions of the CLT.”
DKM: Differentiable K-Means Clustering Layer for Neural Network Compression (2021, methodology): “, 2016), LR-Net (Shayer et al., 2018), and Meta-Quant (Chen et al.”
A Survey on Methods and Theories of Quantized Neural Networks (2018, background): “In [Shayar et al. , 2017], the authors assumed that each weight is sampled independently from a multinomial distribution and the loss function of the network is as follows, This function is not differentiable due to the discreteness.”
And the Bit Goes Down: Revisiting the Quantization of Neural Networks (2019, methodology): “…benchmark for ResNet-18 and ResNet-50 architectures and compare our results to the following methods: Trained Ternary Quantization (TTQ) [23], LR-Net [22], ABC-Net [26], Binary Weight Network (XNOR-Net or BWN) [25], Deep Compression (DC) [15] and Hardware-Aware Automated Quantization (HAQ) [41].”
Bayesian Bits: Unifying Quantization and Pruning (2020, methodology): “Full precision 32/32 69.68 100 (Jacob et al., 2018) 8/8 70.38 6.25 TWN (Li et al., 2017a) 2/32 61.80 5.95 LR-Net (Shayer et al., 2017)* 1/32 59.90 4.58 XNOR-net (Rastegari et al., 2016) 1/1 51.20 0.14 RQ (Louizos et al., 2019) 5/5 65.10 2.54 PACT* (Choi et al., 2018) 4/4 69.20 3.12 PACT* (Choi et…”
Relaxed Quantization for Discretized Neural Networks (2018, unknown): “We compare against multiple works that employ ﬁxed-point quantization: SR+DR (Gupta et al., 2015; Gysel et al., 2018), LR Net (Shayer et al., 2018), Jacob et al. (2017), TWN (Li et al., 2016), INQ (Zhou et al., 2017), BWN (Rastegari et al., 2016), XNOR-net (Rastegari et al., 2016), DoReFa (Zhou et…”
Entropy-Constrained Training of Deep Neural Networks (2018, methodology): “Hence, our work can also be seen as a generalization, which allows for arbitrary cardinality of the discrete set and adds an entropy term to the cost, which prioritizes non-uniform weight distributions as opposed to the training objectives stated in [32], [33].”