LightNN: Filling the Gap between Conventional Deep Neural Networks and Binarized Networks

By Ruizhou Ding, Zeye (Dexter) Liu, Rongye Shi, Diana Marculescu, R. D. Blanton. 2017

38 citations recorded by the source corpus.

Application-specific integrated circuit (ASIC) implementations for Deep Neural Networks (DNNs) have been adopted in many systems because of their higher classification speed. However, although they may be characterized by better accuracy, larger DNNs require significant energy and area, thereby limiting their wide adoption. The energy consumption of DNNs is driven by both memory accesses and computation. Binarized Neural Networks (BNNs), as a trade-off between accuracy and energy consumption, can achieve great energy reduction, and have good accuracy for large DNNs due to its regularization effect. However, BNNs show poor accuracy when a smaller DNN configuration is adopted. In this paper, we propose a new DNN model, LightNN, which replaces the multiplications to one shift or a constrained number of shifts and adds. For a fixed DNN configuration, LightNNs have better accuracy at a slight energy increase than BNNs, yet are more energy efficient with only slightly less accuracy than conventional DNNs. Therefore, LightNNs provide more options for hardware designers to make trade-offs between accuracy and energy. Moreover, for large DNN configurations, LightNNs have a regularization effect, making them better in accuracy than conventional DNNs. These conclusions are verified by experiment using the MNIST and CIFAR-10 datasets for different DNN configurations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jun Du
2. Jaewon Lee
3. Zirui Wang
4. Jae-Eon Jo
5. Hanhwi Jang
6. Jang-Hyun Kim
7. Joonsung Kim

In Computer Vision and Pattern Recognition

1. Jun Du
2. Zirui Wang

In Electrical and Electronic Engineering

1. Jun Du
2. Zirui Wang

Citations in the corpus

• Joint architecture and knowledge distillation in CNN for Chinese text recognition (2019, background): “Different from constraining the weights to +1 or − 1, Ding et al. [38,39] replaced the multiply-accumulate operation with one shift or a constrained number of shifts and adds, which can make trade-offs between accuracy and computational consumption.”
• MnnFast: A Fast and Scalable System Architecture for Memory-Augmented Neural Networks (2019, background): “[16] replace expensive multipliers with cheaper shifters and adders.”