HAQ: Hardware-Aware Automated Quantization

By Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, Song Han. 2018

101 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware (2018, neutral): “Therefore, it is a multi-objective NAS task (Hsu et al., 2018; Dong et al., 2018; Elsken et al., 2018a; He et al., 2018; Wang et al., 2018; Tan et al., 2018), where one of the objectives is non-differentiable (i.e. latency).”
TSM: Temporal Shift Module for Efficient Video Understanding (2018, background): “Another way is to prune, quantize and compress an existing model for efficient deployment [16, 15, 29, 59, 18, 47].”
The hardware lottery (2020, background): “It is a reasonable prediction that the next few generations of chips or specialized kernels will correct for hardware biases against these techniques (Wang et al., 2018; Sun et al., 2020).”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “In HAWQv2, this method was extended to mixed-precision activation quantization [50], and was shown to be more than 100x faster than RL based mixed-precision methods [238].”
I-BERT: Integer-only BERT Quantization (2021, neutral): “These techniques can be broadly categorized into (1) model pruning [18, 31, 35, 38, 40, 67], (2) knowledge distillation [21, 39, 43, 49, 70], (3) efficient neural architecture design [23, 24, 37, 51, 57], (4) hardware and neural architecture co-design [16, 17, 22, 29, 64], and (5) quantization [5, 7”
SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers (2019, unknown): “Advances include depth-wise separable layers [50], deployment-centric pruning [62, 45], quantization [58, 21], and matrix decomposition techniques [55].”
LSQ+: Improving low-bit quantization through learnable offsets and better initialization (2020, background): “This sensitivity problem is ampliﬁed in the presence of depthwise separable convolutions which are known to be challenging to quantize [30].”
Low-Memory Neural Network Training: A Technical Report (2019, methodology): “More recent techniques based on similar principles include (Dubey et al., 2018, Reagen et al., 2018, Wang et al., 2018, Wen et al., 2016).”
Tiny Transfer Learning: Towards Memory-Efficient On-Device Learning (2020, background): “Starting from [10,14,18, 19,44], one line of research focuses on compressing pre-trained neural networks, including i) network pruning that removes less-important units [12,19] or channels [20,33]; ii) network quantization that reduces the bitwidth of parameters [7,18] or activations [26,45].”
HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks (2019, methodology): “We also compare with the recent AutoML approach of [22].”
HAWQV3: Dyadic Neural Network Quantization (2020, methodology): “HAQ [50] proposes to search this space by applying a Reinforcement Learning (RL) algorithm, while [52] uses a Differentiable Neural Architecture Search (DNAS) to accelerate this searching process.”
Towards Unified INT8 Training for Convolutional Neural Network (2019, background): “As a result, many recent studies have been focusing on how to accelerate the inference of neural networks by fixed-point quantization on weights or activations [6, 3, 24, 29, 27, 64, 42, 52, 63, 54, 21, 44, 57], and design dedicated hardware utilizing the efficient integer arithmetic [17, 5, 26, 23]”
TinyTL: Reduce Memory, Not Parameters for Efficient On-Device Learning (2020, background): “14], one line of research focuses on compressing pre-trained neural networks, including i) network pruning that removes less-important units [4, 15] or channels [16, 17]; ii) network quantization that reduces the bitwidth of parameters [5, 18] or activations [19, 20].”
Mixed Precision DNNs: All you need is a good parametrization (2019, background): “One approach was proposed in (Wang et al., 2018; Elthakeb et al., 2018).”
ReLeQ : A Reinforcement Learning Approach for Automatic Deep Quantization of Neural Networks (2020, methodology): “There is a concurrent work HAQ, 12 which also uses RL in the context of quantization.”
Memory-Driven Mixed Low Precision Quantization For Enabling Deep Network Inference On Microcontrollers (2019, background): “On the same direction, HAQ [22] dynamically explores multiple low-bitwidth precision at training time by means of reinforcement learning.”