Binarized Neural Networks

By Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio. 2016

2,178 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Mixed Precision Training (2017, background): “Hubara et al. (2016a) extended that work to binarize both weights and activations, but gradients were stored and computed in single precision.”
Dynamic Neural Networks: A Survey (2021, methodology): “ation techniques for staticmodels [28], [29], [44], which handle ”easy” and ”hard” inputswith identical computation, and fail to reduce intrinsic computational redundancy.”
Why Should We Add Early Exits to Neural Networks? (2020, background): “More in general, several authors have considered lowering the implementation complexity by reducing the computational precision of the performed per-layer operations, either through compression (e.g., group sparsity), quantization, or the introduction of speciﬁc low-latency components [ 26, 59 ] .”
Expanding the Reach of Federated Learning by Reducing Client Resource Requirements (2018, methodology): “Popular alternatives include pruning the least useful connections in a network [Han et al., 2016, 2015], weight quantization [Hubara et al., 2016, Lin et al., 2017, De Sa et al., 2018], and model distillation [Hinton et al., 2015].”
QSGD: Communication-Optimal Stochastic Gradient Descent, with Applications to Training Neural Networks (2016, background): “Proposals such as [37, 14, 18] investigate deterministic and stochastic quantization for weights, activations, and gradients during training, with as little as one bit per dimension.”
FBNetV3: Joint Architecture-Recipe Search using Neural Acquisition Function (2020, background): “Non-architectural modifications: This involves a number of network compression techniques including low-bit quantization [9] to as few as two [49] or even one bit [17].”
Learning to Prune Filters in Convolutional Neural Networks (2018, methodology): “[16] introduced an algorithm to train networks with binary weights and activations.”
Fast Lossless Neural Compression with Integer-Only Discrete Flows (2022, background): “Quantization represents parameters and activations in low precision format instead of 32-bit floating-point numbers (Courbariaux et al., 2015; Hubara et al., 2016; Rastegari et al., 2016; Zhou et al., 2016; Jacob et al., 2018; Choi et al., 2018; Dong et al., 2019; Esser et al., 2020; Chen et al., 20”
PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning (2017, methodology): “In our current approach, the sparsity masks per task are obtained as a result of pruning, but it might be possible to learn such masks using techniques similar to those for learning networks with binary weights [12, 23].”
TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning (2017, background): “This stochastic rounding, instead of deterministic one, is chosen by both our study and QSGD [29], as stochastic rounding has an unbiased expectation and has been successfully studied for low-precision processing [20][30].”
CondenseNet: An Efficient DenseNet Using Learned Group Convolutions (2017, background): “CondenseNets are closely related to approaches that improve the inference efficiency of (convolutional) networks via weight pruning [11, 14, 27, 29, 32] and/or weight quantization [21, 36].”
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference (2017, methodology): “With these approaches, both multiplications and additions can be implemented by efficient bit-shift and bit-count operations, which are showcased in custom GPU kernels (BNN [14]).”
LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks (2018, methodology): “Representative techniques include quantization [6, 34, 29, 54, 52, 21, 39, 53, 22, 3, 55, 9, 18], pruning [13, 12, 17, 36], low-rank decomposition [7, 8, 24, 27, 38, 51, 49], hashing [4], and deliberate architecture design [23, 19, 50].”
Capacity and Trainability in Recurrent Neural Networks (2016, background): “…previous results on model compression – by reducing the number of parameters (Yang et al., 2015), or by reducing the bit depth of parameters (Hubara et al., 2016) – might lead one to predict that different architectures use parameters with vastly different efﬁciencies, and that task…”
Adaptive Neural Networks for Efficient Inference (2017, background): “Reduced precision networks (Gong et al., 2014; Cour-bariaux et al., 2015; Chen et al., 2015; Hubara et al., 2016a; Wu et al., 2016; Rastegari et al., 2016; Hubara et al., 2016b) have been extensively studied to reduce the memory foot-print of networks and their test-time cost.”
ChamNet: Towards Efficient Network Design Through Platform-Aware Model Adaptation (2018, methodology): “Quantization [10, 17] has also emerged as a powerful tool for significantly cutting down computation cost with no or little accuracy loss.”
SNIP: Single-shot Network Pruning based on Connection Sensitivity (2018, background): “A non-exhaustive list includes quantization (Gong et al. (2014)), reduced precision (Gupta et al. (2015)) and binary weights (Hubara et al. (2016)).”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Bi-narized NN [106] (BNN) extends this idea by binarizing the activations as well as the weights.”