Accelerating Convolutional Neural Networks via Activation Map Compression

By Georgios Georgiadis. 2018

86 citations recorded by the source corpus.

The deep learning revolution brought us an extensive array of neural network architectures that achieve state-of-the-art performance in a wide variety of Computer Vision tasks including among others, classification, detection and segmentation. In parallel, we have also been observing an unprecedented demand in computational and memory requirements, rendering the efficient use of neural networks in low-powered devices virtually unattainable. Towards this end, we propose a three-stage compression and acceleration pipeline that sparsifies, quantizes and entropy encodes activation maps of Convolutional Neural Networks. Sparsification increases the representational power of activation maps leading to both acceleration of inference and higher model accuracy. Inception-V3 and MobileNet-V1 can be accelerated by as much as 1.6x with an increase in accuracy of 0.38% and 0.54% on the ImageNet and CIFAR-10 datasets respectively. Quantizing and entropy coding the sparser activation maps lead to higher compression over the baseline, reducing the memory cost of the network execution. Inception-V3 and MobileNet-V1 activation maps, quantized to 16 bits, are compressed by as much as 6x with an increase in accuracy of 0.36% and 0.55% respectively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks (2022, background): “Prior work has observed that using ReLUs leads to a per-layer activation distribution that is highly skewed across different network architectures and datasets (Kurtz et al., 2020; Hu et al., 2016; Rhu et al., 2018; Georgiadis, 2019).”
Latent Replay for Real-Time Continual Learning (2019, methodology): “This can be easily achieved by adding an L1 term (with relative weight) to the loss function attracting toward zero the activations of the latent replay layer (see [1]).”
Sparse Autoencoders Find Highly Interpretable Features in Language Models (2023, methodology): “These approaches include altering the attention mechanism (Correia et al., 2019), adding ℓ 1 penalties to neuron activations (Kasioumis et al., 2021; Georgiadis, 2019), pruning neurons (Frankle & Carbin, 2018), and using the softmax function as the non-linearity in the MLP layers (Elhage et al.,…”
Inducing and Exploiting Activation Sparsity for Fast Neural Network Inference (2020, methodology): “The work closest to ours is (Georgiadis, 2019), who proposed and investigated the use of L1-regularization applied to the activation maps, and showed that it can result in a signiﬁcant increase (up to 60% relative to naturally-occurring activation sparsity) on a range of CNNs for image…”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “Some key distinctions are drawn between global versus local interpretability and comprehensive versus partial interpretability.”
Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation (2025, unknown): “…or training protocols, including modifications to attention mechanisms (Correia et al., 2019), applying Bayesian standard Gamma priors (Duan et al., 2024), incorporating discrete sparse concept layers (Koh et al., 2020), and promoting sparse activations in large language models (Mirzadeh et al.,…”