Binarized Neural Networks

By Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio. 2016

1,334 citations recorded by the source corpus.

We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time. At train-time the binary weights and activations are used for computing the parameter gradients. During the forward pass, BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which is expected to substantially improve power-efficiency. To validate the effectiveness of BNNs, we conducted two sets of experiments on the Torch7 and Theano frameworks. On both, BNNs achieved nearly state-of-the-art results over the MNIST, CIFAR-10 and SVHN datasets. We also report our preliminary results on the challenging ImageNet dataset. Last but not least, we wrote a binary matrix multiplication GPU kernel with which it is possible to run our MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The code for training and running our BNNs is available on-line.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Sequence-Level Knowledge Distillation (2016, background): “…of weight matrices (Denton et al., 2014; Jaderberg et al., 2014; Lu et al., 2016; Prabhavalkar et al., 2016), sparsity-inducing regularizers (Murray and Chiang, 2015), binarization of weights (Courbariaux et al., 2016; Lin et al., 2016), and weight sharing (Chen et al., 2015; Han et al., 2016).”
Mixed Precision Training (2017, background): “Hubara et al. (2016b) propose another approach to quantize RNNs without altering their structure.”
FastText.zip: Compressing text classification models (2016, background): “Some use vector quantization (Gong et al., 2014) while others binarize the network (Courbariaux et al., 2016).”
Hierarchical Multiscale Recurrent Neural Networks (2016, methodology): “We ﬁnd that the straight-through estimator (Hinton, 2012; Bengio et al., 2013; Courbariaux et al., 2016) is efﬁcient for training this model containing discrete variables.”
Distilling Task-Specific Knowledge from BERT into Simple Neural Networks (2019, methodology): “Still other studies examine quantizing neural networks (Wu et al., 2018); in the extreme, Courbariaux et al. (2016) propose binarized networks with both binary weights and binary activations.”
Secure Evaluation of Quantized Neural Networks (2019, methodology): “Garbled Circuits, use a quantization scheme which converts weights into bits [34].”
Learning Multimodal Graph-to-Graph Translation for Molecular Optimization (2018, methodology): “…d ( i t , j t ) , to account for the fact that the messages should depend on the topological layout of the tree: As d ( i t , j t ) is computed by a non-differentiable threshold function, we approximate its gradient with a straight-through estimator (Bengio et al., 2013; Courbariaux et al., 2016).”
Differentiable Model Compression via Pseudo Quantization Noise (2021, methodology): “Related Work Early network quantization methods focused on low-bitwidth networks such as BinaryNet (Courbariaux, Bengio, and David 2015; Courbariaux et al. 2016), XOR-Nets (Rastegari et al.”
Dynamic Network Surgery for Efficient DNNs (2016, background): “The recently proposed BinaryConnect [2] and Binarized Neural Networks [3] are able to compress DNNs by a factor of 32×, while a noticeable accuracy loss is sort of inevitable.”