BinaryConnect: Training Deep Neural Networks with binary weights during propagations
By Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre David. 2015
3,245 citations recorded by the source corpus.
Deep Neural Networks (DNN) have achieved state-of-the-art results in a wide range of tasks, with the best results obtained with large training sets and large models. In the past, GPUs enabled these breakthroughs because of their greater computational speed. In the future, faster computation at both training and test time is likely to be crucial for further progress and for consumer applications on low-power devices. As a result, there is much interest in research and development of dedicated hardware for Deep Learning (DL). Binary weights, i.e., weights which are constrained to only two possible values (e.g. -1 or 1), would bring great benefits to specialized DL hardware by replacing many multiply-accumulate operations by simple accumulations, as multipliers are the most space and power-hungry components of the digital implementation of neural networks. We introduce BinaryConnect, a method which consists in training a DNN with binary weights during the forward and backward propagations, while retaining precision of the stored weights in which gradients are accumulated. Like other dropout schemes, we show that BinaryConnect acts as regularizer and we obtain near state-of-the-art results with BinaryConnect on the permutation-invariant MNIST, CIFAR-10 and SVHN.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Quoc V. Le
- Mohit Bansal
- Dacheng Tao
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Yi Yang
- Jitendra Malik
- Alan L. Yuille
In Efficiency & Compression
- Song Han
- Dan Alistarh
- Kurt Keutzer
- Tijmen Blankevoort
- Yunhe Wang
- Zi-Mei Yao
- Quoc V. Le
- Mohammad Rastegari
Citations in the corpus
- Mixed Precision Training (2017, background): “Courbariaux et al. (2015) proposed training with binary weights, but all other tensors and arithmetic were in full precision.”
- Large Memory Layers with Product Keys (2019, neutral): “Discretization techniques have been intensively studied for compressing network weights [8, 36] and/or activations [7, 36] or to accelerate inference.”
- Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “Some methods approximate model weights using quantization and hashing [Gong et al., 2014, Courbariaux et al., 2015], or low-rank approximation and tensor factorization [Denil et al., 2013, Lebedev et al., 2015, Su et al., 2018].”
- Stable and low-precision training for large-scale vision-language models (2023, background): “One major direction is to quantize convolutional neural networks often for fast and memory efficient usage on edge devices [78, 6, 13, 79, 23, 77, 30].”
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “Methods for different bit-widths have been studied: 1-bit methods (Courbariaux and Bengio, 2016; Rastegari et al., 2016; Courbariaux et al., 2015), 2 to 3-bit (Zhu et al., 2017; Choi et al., 2019), 4-bits (Li et al., 2019), more bits (Courbariaux et al., 2014), or a variable amount of bits (Gong et…”
- 8-bit Optimizers via Block-wise Quantization (2021, methodology): “There are methods that use 1-bit (Courbariaux and Bengio, 2016; Rastegari et al., 2016; Courbariaux et al., 2015), 2-bit/3 values (Zhu et al.”
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, methodology): “Courbariaux et al. (2015b) similarly used two sets of weights, real-valued and binary.”
- Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT (2019, neutral): “Here we solely focus on quantization [4, 6, 8, 14, 17, 24, 40, 42].”
- XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks (2016, methodology): “We compare our Binary-weight-network ( BWN ) with BinaryConnect ( BC ) [38] and our XNOR-Networks ( XNOR-Net ) with BinaryNeuralNet ( BNN ) [11].”
- Hierarchical Multiscale Recurrent Neural Networks (2016, methodology): “per operation at each time step and to detect the boundaries, the HM-RNN discovers the latent hierarchical structure of the sequences. We find that the straight-through estimator (Bengio et al., 2013; Courbariaux et al., 2015) is efficient for training this model containing discrete variables. We eval”
- Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations (2016, methodology): “, 2016) learn update probabilities for different units using the straight-through estimator (Bengio et al., 2013; Courbariaux et al., 2015), and combined with recently-proposed Layer Normalization (Ba et al.”
- Advances and Open Problems in Federated Learning (2019, background): “This topic has received a lot of attention in the broader ML community [194, 120, 436, 270, 312], but these methods either do not have a straightforward mapping to federated learning, or make the training process more complex which makes it difficult to adopt.”
- AutoML: A Survey of the State-of-the-Art (2019, methodology): “are binarized using the BinaryConnect method [98] and hence only one path is activated at training time.”
- Secure Evaluation of Quantized Neural Networks (2019, methodology): “…1990s [23, 4, 61, 48], and this extensive research body have enabled modern quantized neural networks to have essentially the same accuracy as their full-precision counterparts [12, 68, 26, 30, 52], even with large CNN architectures like AlexNet [43], VGGNet [59], GoogleNet [60] and ResNet [31].”
- To prune, or not to prune: exploring the efficacy of pruning for model compression (2017, background): “In addition, an emerging area of research is low precision networks where the parameters and/or activations are quantized to 4 bits or fewer (Courbariaux et al., 2015; Lin et al., 2015; Hubara et al., 2016; Rastegari et al., 2016; Zhu et al., 2016).”
- Dynamic Model Pruning with Feedback (2020, methodology): “One variant of this approach is implemented in the Binary Connect algorithm (BC) (Courbariaux et al., 2015) with prominent results, see also (Li et al., 2017) for further insights and discussion.”
- Paraphrasing Complex Network: Network Compression via Factor Transfer (2018, background): “There are also many studies that directly train convolution neural network (CNN) using binary weights (Courbariaux et al., 2015; 2016; Rastegari et al., 2016).”
- Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness Against Adversarial Attack (2018, background): “ing model from overfiting the training set. Traditional regularization methods have been effective in neural network training. For example, dropout [25], Batch Normalization (BN) [26] and quantization [27, 28] all serve the purpose of model regularization. However, BN is specifically effective in conv”
- Model compression via distillation and quantization (2018, background): “The first direction is the work on training quantized neural networks, e.g. Courbariaux et al. (2015); Rastegari et al. (2016); Hubara et al. (2016); Wu et al. (2016a); Mellempudi et al. (2017); Ott et al. (2016); Zhu et al. (2016), which showed that neural networks can converge to good task…”
- AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, neutral): “The extreme case could even just use binary values (Courbariaux et al., 2015; Rastegari et al., 2016).”