BinaryNet: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1

By Matthieu Courbariaux, Yoshua Bengio. 2016

704 citations recorded by the source corpus.

Subfields: Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “Methods for different bit-widths have been studied: 1-bit methods (Courbariaux and Bengio, 2016; Rastegari et al., 2016; Courbariaux et al., 2015), 2 to 3-bit (Zhu et al., 2017; Choi et al., 2019), 4-bits (Li et al., 2019), more bits (Courbariaux et al., 2014), or a variable amount of bits (Gong et…”
8-bit Optimizers via Block-wise Quantization (2021, methodology): “There are methods that use 1-bit (Courbariaux and Bengio, 2016; Rastegari et al., 2016; Courbariaux et al., 2015), 2-bit/3 values (Zhu et al.”
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks (2016, methodology): “Our experimental results show that our proposed method for binarizing convolutional neural networks outperforms the state-of-the-art network binarization method of [11] by a large margin (16 .”
Pruning Filters for Efficient ConvNets (2016, methodology): “Additionally, quantization (Han et al. (2016b)) and binarization (Rastegari et al. (2016); Courbariaux & Bengio (2016)) can be used to reduce the model size and lower the computation overheads.”
Recent advances in convolutional neural networks (2015, background): “[150] describes how to train fully-connected networks and CNNs with full binarization and batch normalization layers, reporting competitive accuracies on the MNIST, SVHN, and CIFAR-10 datasets.”
AMC: AutoML for Model Compression and Acceleration on Mobile Devices (2018, background): “Quantization [55,10,41] and special convolution implementations [36,48,29,3] can also speed up the neural networks.”
Learning Efficient Convolutional Networks through Network Slimming (2017, methodology): “These include low-rank approximation [7], network quantization [3, 12] and binarization [28, 6], weight pruning [12], dynamic inference [16], etc.”
Bayesian Compression for Deep Learning (2017, methodology): “As an extreme example, [13, 48, 57, 76] and [11] trained NN to use only binary or tertiary weights with floating point gradients.”
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients (2016, background): “In particular, in BNN (Cour-bariaux & Bengio, 2016) and XNOR-Net (Rastegari et al., 2016), both weights and input activations of convolutional layers 1 are binarized.”
Soft Weight-Sharing for Neural Network Compression (2017, background): “…for a number of highly successful compression algorithms, which either train the network from scratch (Hinton et al., 2015; Iandola et al., 2016; Courbariaux & Bengio, 2016; Courbariaux et al., 2016) or apply compression post-optimization (Han et al., 2015b;a; Guo et al., 2016; Chen et al.,…”
Training with Quantization Noise for Extreme Model Compression (2020, methodology): “STE introduces a bias in the gradients that depends on level of quantization of the weights, and thus, the compression ratio.”
Demystifying Parallel and Distributed Deep Learning (2018, background): “Binarized Neural Networks (BNNs) [50], Ternary Weight Networks [156], TernGrad [245], and DoReFa-Net [276] quantize networks to binary parameters, ternary parameters, ternary gradients, and binary parameters+ternary gradients, respectively.”
More is Less: A More Complicated Network with Less Inference Complexity (2017, methodology): “In [2, 3], a standard network with binary weights represented by 1-bit was trained to speed up networks.”
NIPQ: Noise Injection Pseudo Quantization for Automated DNN Optimization (2022, background): “Thus, not only early studies [11, 12, 13] but also advanced ones [14, 15, 16] have proposed diverse QAT schemes based on STE and shown that complex neural networks (i.e., ResNet-18 [17]) can be quantized into 4-bit without accuracy loss.”
WRPN: Wide Reduced-Precision Networks (2017, background): “XNOR-NET [17], BNN [4], DoReFa [26] and TTQ [27] target training as well.”
Entropy-Constrained Training of Deep Neural Networks (2018, background): “Others, on reducing the cardinality of the networks parameters [28]–[33].”