BinaryConnect: Training Deep Neural Networks with binary weights during propagations

By Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre David. 2015

3,245 citations recorded by the source corpus.

Deep Neural Networks (DNN) have achieved state-of-the-art results in a wide range of tasks, with the best results obtained with large training sets and large models. In the past, GPUs enabled these breakthroughs because of their greater computational speed. In the future, faster computation at both training and test time is likely to be crucial for further progress and for consumer applications on low-power devices. As a result, there is much interest in research and development of dedicated hardware for Deep Learning (DL). Binary weights, i.e., weights which are constrained to only two possible values (e.g. -1 or 1), would bring great benefits to specialized DL hardware by replacing many multiply-accumulate operations by simple accumulations, as multipliers are the most space and power-hungry components of the digital implementation of neural networks. We introduce BinaryConnect, a method which consists in training a DNN with binary weights during the forward and backward propagations, while retaining precision of the stored weights in which gradients are accumulated. Like other dropout schemes, we show that BinaryConnect acts as regularizer and we obtain near state-of-the-art results with BinaryConnect on the permutation-invariant MNIST, CIFAR-10 and SVHN.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yoshua Bengio
  2. Quoc V. Le
  3. Mohit Bansal
  4. Dacheng Tao
  5. Jianfeng Gao
  6. Percy Liang
  7. Luke S. Zettlemoyer
  8. Ruslan Salakhutdinov

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Ming-Hsuan Yang
  6. Yi Yang
  7. Jitendra Malik
  8. Alan L. Yuille

In Efficiency & Compression

  1. Song Han
  2. Dan Alistarh
  3. Kurt Keutzer
  4. Tijmen Blankevoort
  5. Yunhe Wang
  6. Zi-Mei Yao
  7. Quoc V. Le
  8. Mohammad Rastegari

Citations in the corpus