Fixed-point feedforward deep neural network design using weights +1, 0, and −1

By Kyuyeon Hwang, Wonyong Sung. 2014

282 citations recorded by the source corpus.

Feedforward deep neural networks that employ multiple hidden layers show high performance in many applications, but they demand complex hardware for implementation. The hardware complexity can be much lowered by minimizing the word-length of weights and signals, but direct quantization for fixed-point network design does not yield good results. We optimize the fixed-point design by employing backpropagation based retraining. The designed fixed-point networks with ternary weights (+1, 0, and -1) and 3-bit signal show only negligible performance loss when compared to the floating-point coun-terparts. The backpropagation for retraining uses quantized weights and fixed-point signal to compute the output, but utilizes high precision values for adapting the networks. A character recognition and a phoneme recognition examples are presented.

Subfields: Artificial Intelligence, Computational Mechanics, Model Architectures, Efficiency & Compression, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Model Architectures

In Efficiency & Compression

In Signal Processing

Citations in the corpus

Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, background): “Hwang and Sung (2014) focused on a fixed-point neural network design and achieved performance almost identical to that of the floating-point architecture.”
Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding (2015, methodology): “Hwang & Sung (2014) proposed an optimization method for the ﬁxed-point network with ternary weights and 3-bit activations.”
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks (2016, methodology): “Another ﬁxed-point network with ternary weights and 3-bits activations was presented by [32].”
BinaryConnect: Training Deep Neural Networks with binary weights during propagations (2015, background): “d we train for 200 epochs instead of 500 (because SVHN is quite a big dataset). The results are in Table 2. 4 Related works Training DNNs with binary weights has been the subject of very recent works [37, 38, 39, 40]. Even though we share the same objective, our approaches are quite different. [37, ”
Bitwise Neural Networks (2016, methodology): “A special part of this network is that we constrain the weights to have values between −1 and +1 as well by wrapping them with tanh.”
Training Quantized Nets: A Deeper Understanding (2017, background): “Previous work on obtaining a quantized neural network can be divided into two categories: quantizing pre-trained models with or without retraining [7, 13, 6, 14, 15], and training a quantized model from scratch [4, 5, 3, 1, 16].”
Accurate and Efficient 2-bit Quantized Neural Networks (2019, background): “One of the earliest studies in weight quantization schemes (Hwang & Sung, 2014; Courbariaux et al., 2015) show that it is indeed possible to quantize weights to 1-bit (binary) or 2-bits (ternary), allowing an entire DNN model to ﬁt in resource-constrained platforms (e.g., mobile devices).”
Reduced-Precision Strategies for Bounded Memory in Deep Neural Nets (2015, methodology): “Gupta et al. (2015) trains neural networks with 16-bit fixed-point numbers and stochastic rounding.”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, background): “• High precision bias [108], first and last layers [190, 261] could be reserved to maintain network accuracy.”
BinaryNet: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1 (2016, background): “Hwang & Sung (2014); Kim et al. (2014) retrain neural networks with ternary weights and 3-bit activations, i.e.: • They train a neural network with high-precision,”
LCNN: Lookup-Based Convolutional Neural Network (2016, methodology): “Low Precision Networks: A fixed-point implementation of 8-bit integer was compared with 32-bit floating point activations in [41, 20].”
A Survey on Methods and Theories of Quantized Neural Networks (2018, background): “, 2016] Fixed codebook quantization Ternarization {-1, 0, 1} [Hwang and Sung, 2014] Scaled Ternarization {-a,0,b} [Zhu et al.”