Heterogeneous Bitwidth Binarization in Convolutional Neural Networks

By Josh Fromm, Shwetak N. Patel, Matthai Philipose. 2018

29 citations recorded by the source corpus.

Recent work has shown that fast, compact low-bitwidth neural networks can be surprisingly accurate. These networks use homogeneous binarization: all parameters in each layer or (more commonly) the whole model have the same low bitwidth (e.g., 2 bits). However, modern hardware allows efficient designs where each arithmetic instruction can have a custom bitwidth, motivating heterogeneous binarization, where every parameter in the network may have a different bitwidth. In this paper, we show that it is feasible and useful to select bitwidths at the parameter granularity during training. For instance a heterogeneously quantized version of modern networks such as AlexNet and MobileNet, with the right mix of 1-, 2- and 3-bit parameters that average to just 1.4 bits can equal the accuracy of homogeneous 2-bit versions of these networks. Further, we provide analyses to show that the heterogeneously binarized systems yield FPGA- and ASIC-based implementations that are correspondingly more efficient in both circuit area and energy efficiency than their homogeneous counterparts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Flower: A Friendly Federated Learning Research Framework (2020, background): “…has been tremendous progress in enabling the execution of deep learning models on mobile and embedded devices to infer user contexts and behaviors (Fromm et al., 2018; Chowdhery et al., 2019; Malekzadeh et al., 2019; Lee et al., 2019; Yao et al., 2019; LiKamWa et al., 2016; Georgiev et al., 2017).”
Similarity-Preserving Knowledge Distillation (2019, background): “For example, unstructured pruning requires optimized sparse matrix multiplication routines to realize practical acceleration [26], platform constraint-aware compression [2, 36, 37] requires hardware simulators or empirical measurements, and arbitrarybit quantization [9, 17] requires specialized hard”
Memory-Driven Mixed Low Precision Quantization For Enabling Deep Network Inference On Microcontrollers (2019, methodology): “The method (Fromm et al., 2018) targeted perpixel binarization based on a defined tensor mask.”