XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
By Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, Ali Farhadi. 2016
4,754 citations recorded by the source corpus.
We propose two efficient approximations to standard convolutional neural networks: Binary-Weight-Networks and XNOR-Networks. In Binary-Weight-Networks, the filters are approximated with binary values resulting in 32\(\times \) memory saving. In XNOR-Networks, both the filters and the input to convolutional layers are binary. XNOR-Networks approximate convolutions using primarily binary operations. This results in 58\(\times \) faster convolutional operations (in terms of number of the high precision operations) and 32\(\times \) memory savings. XNOR-Nets offer the possibility of running state-of-the-art networks on CPUs (rather than GPUs) in real-time. Our binary networks are simple, accurate, efficient, and work on challenging visual tasks. We evaluate our approach on the ImageNet classification task. The classification accuracy with a Binary-Weight-Network version of AlexNet is the same as the full-precision AlexNet. We compare our method with recent network binarization methods, BinaryConnect and BinaryNets, and outperform these methods by large margins on ImageNet, more than \(16\,\%\) in top-1 accuracy. Our code is available at: http://allenai.org/plato/xnornet.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Quoc V. Le
- Dacheng Tao
- Yoshua Bengio
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Trevor Darrell
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Yi Yang
- Xiaogang Wang
- Ming-Hsuan Yang
- Trevor Darrell
- Jitendra Malik
- Alan L. Yuille
In Efficiency & Compression
Citations in the corpus
- Experience Grounds Language (2020, background): “These advances have lead to parameter efficient models (Tan and Le, 2019) and real time object detection on embedded devices and phones (Rastegari et al., 2016; Redmon and Farhadi, 2018).”
- Mixed Precision Training (2017, neutral): “Rastegari et al. (2016) binarize all tensors, including the gradients.”
- Large Memory Layers with Product Keys (2019, neutral): “Discretization techniques have been intensively studied for compressing network weights [8, 38] and/or activations [7, 38] or to accelerate inference.”
- Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, neutral): “Early work in this direction includes [23, 31, 13, 17, 32, 16, 48, 27, 43].”
- The case for 4-bit precision: k-bit Inference Scaling Laws (2022, methodology): “For example, there are quantization methods where a data type is optimized with additional input data (Rastegari et al., 2016; Frantar et al., 2022) or from the weights of the model alone (Gong et al., 2014).”
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “Methods for different bit-widths have been studied: 1-bit methods (Courbariaux and Bengio, 2016; Rastegari et al., 2016; Courbariaux et al., 2015), 2 to 3-bit (Zhu et al., 2017; Choi et al., 2019), 4-bits (Li et al., 2019), more bits (Courbariaux et al., 2014), or a variable amount of bits (Gong et…”
- 8-bit Optimizers via Block-wise Quantization (2021, methodology): “There are methods that use 1-bit (Courbariaux and Bengio, 2016; Rastegari et al., 2016; Courbariaux et al., 2015), 2-bit/3 values (Zhu et al., 2017; Choi et al., 2019), 4-bits (Li et al., 2019), more bits (Courbariaux et al., 2014), or a variable amount of bits (Gong et al., 2019).”
- Why Should We Add Early Exits to Neural Networks? (2020, background): “More in general, several authors have considered lowering the implementation complexity by reducing the computational precision of the performed per-layer operations, either through compression (e.g., group sparsity), quantization, or the introduction of specific low-latency components [ 26, 59 ] .”
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, methodology): “Moreover, (Rastegari et al., 2016) didn’t quantize first and last layers, therefore XNOR-Net are only partially binarized NNs. Miyashita et al. (2016) suggested a more relaxed quantization (more than 1-bit) for both the weights and activation.”
- Green AI (2019, background): “For example, a significant amount of work in the computer vision community has addressed efficient inference, which is necessary for real-time processing of images for applications like self-driving cars [23, 30, 22], or for placing models on devices such as mobile phones [16, 33].”
- ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices (2017, background): “Note that many existing works [15, 20, 39, 38, 35, 24] focus on pruning, compressing, or low-bit representing a “basic" network architecture.”
- Pruning Filters for Efficient ConvNets (2016, methodology): “Additionally, quantization (Han et al. (2016b)) and binarization (Rastegari et al. (2016); Courbariaux & Bengio (2016)) can be used to reduce the model size and lower the computation overheads.”
- Quantized Convolutional Neural Networks for Mobile Devices (2015, background): “Recently, fixed-point based approaches are explored in [5, 25].”
- Secure Evaluation of Quantized Neural Networks (2019, background): “Quantization allows the representation of the weights and activations to be as low as 8 bits, or even 1 bit in some cases [11, 53].”
- QUOTIENT: Two-Party Secure Neural Network Training and Prediction (2019, background): “While previous work [2, 4, 9, 24, 30, 36, 46, 56] had already introduced the idea of functions Qw ,Qa to quantize weights and/or activations in the forward pass, they required the weights W or gradients G, e to be represented in floating-point in the backward pass, in order to optimize accurately.”
- Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware (2018, methodology): “Our work is also orthogonal and complementary to techniques that modify DNNs to speed up inference [21, 30, 42, 27, 15].”
- To prune, or not to prune: exploring the efficacy of pruning for model compression (2017, background): “In addition, an emerging area of research is low precision networks where the parameters and/or activations are quantized to 4 bits or fewer (Courbariaux et al., 2015; Lin et al., 2015; Hubara et al., 2016; Rastegari et al., 2016; Zhu et al., 2016).”