Weightless: Lossy Weight Encoding For Deep Neural Network Compression

By Brandon Reagen, Udit Gupta, Bob Adolf, Michael M. Mitzenmacher, Alexander M. Rush, Gu-Yeon Wei, David M. Brooks. 2017

42 citations recorded by the source corpus.

The large memory requirements of deep neural networks limit their deployment and adoption on many devices. Model compression methods effectively reduce the memory requirements of these models, usually through applying transformations such as weight pruning or quantization. In this paper, we present a novel scheme for lossy weight encoding co-designed with weight simplification techniques. The encoding is based on the Bloomier filter, a probabilistic data structure that can save space at the cost of introducing random errors. Leveraging the ability of neural networks to tolerate these imperfections and by re-training around the errors, the proposed technique, named Weightless, can compress weights by up to 496x without loss of model accuracy. This results in up to a 1.51x improvement over the state-of-the-art.

Subfields: Computer Vision and Pattern Recognition, Efficiency & Compression, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Georgios Georgiadis
2. Udit Gupta
3. David M. Brooks
4. Gu-Yeon Wei
5. Brandon Reagen
6. Alexander M. Rush
7. José Miguel Hernández-Lobato
8. Robert Peharz

In Efficiency & Compression

1. Georgios Georgiadis
2. José Miguel Hernández-Lobato
3. David M. Brooks
4. Gu-Yeon Wei
5. Alexander M. Rush
6. Udit Gupta
7. Brandon Reagen

In Signal Processing

1. Georgios Georgiadis
2. José Miguel Hernández-Lobato
3. Robert Peharz
4. Alexander M. Rush
5. Jang-Hyun Kim

Citations in the corpus

• Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters (2018, methodology): “As baselines we used three recent state-of-the-art methods, namely Deep Compression (Han et al., 2016), Weightless encoding (Reagen et al., 2018) and Bayesian Compression (Louizos et al., 2017).”
• MASR: A Modular Accelerator for Sparse RNNs (2019, background): “Design-time parameters include whether the network is an RNN or GRU, and the maximum recurrent network size supported.”
• Accelerating Convolutional Neural Networks via Activation Map Compression (2018, methodology): “Lossless weight compression has appeared in the literature in the form of Huffman coding (HC) [18] and arithmetic coding [51].”
• MnnFast: A Fast and Scalable System Architecture for Memory-Augmented Neural Networks (2019, background): “[56] propose a novel lossy weight encoding scheme.”