Training with Quantization Noise for Extreme Fixed-Point Compression

21 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, methodology): “[21] introduces quantization noise to alleviate the variations of QAT.”
Efficient Transformers: A Survey (2020, methodology): “(Fan et al., 2020) applies Quantization Aware training to Transformer models.”
Linformer: Self-Attention with Linear Complexity (2020, methodology): “This technique can be further improved through Quantization Aware Training (Jacob et al., 2018; Fan et al., 2020), where the weights are quantized during training and the gradients are approximated with the Straight-Through Estimator.”
Luna: Linear Unified Nested Attention (2021, methodology): “, 2018), quantization (Shen et al., 2020; Fan et al., 2020), sparse attention (Parmar et al.”
I-BERT: Integer-only BERT Quantization (2021, background): “(Fan et al., 2020) quantizes a different subset of weights in each training iteration to make models more robust to quantization.”
MLPruning: A Multilevel Structured Pruning Framework for Transformer-based Models (2021, background): “These efforts can be generally categorized as follows: (i) knowledge distillation [21, 41, 35, 39]; (ii) quantization [3, 54, 37, 12, 53, 55, 2, 10]; (iii) new architecture design [40, 19, 23, 22, 45]; and (iv) pruning.”