Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness

By Young Jin Kim, Raffy Fahim, Hany Hassan Awadalla. 2023

40 citations recorded by the source corpus.

Large Mixture of Experts (MoE) models could achieve state-of-the-art quality on various language tasks, including machine translation task, thanks to the efficient model scaling capability with expert parallelism. However, it has brought a fundamental issue of larger memory consumption and increased memory bandwidth bottleneck at deployment time. In this paper, we propose Mixture of Quantized Experts (MoQE) which is a simple weight-only quantization method applying ultra low-bit down to 2-bit quantizations only to expert weights for mitigating the increased memory and latency issues of MoE models. We show that low-bit quantization together with the MoE architecture delivers a reliable model performance while reducing the memory size significantly even without any additional training in most cases. In particular, expert layers in MoE models are much more robust to the quantization than conventional feedforward networks (FFN) layers. In our comprehensive analysis, we show that MoE models with 2-bit expert weights can deliver better model performance than the dense model trained on the same dataset. As a result of low-bit quantization, we show the model size can be reduced by 79.6% of the original half precision floating point (fp16) MoE model. Combined with an optimized GPU runtime implementation, it also achieves 1.24X speed-up on A100 GPUs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Pulmonary and Respiratory Medicine, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Pulmonary and Respiratory Medicine

In Large Language Models

In Efficiency & Compression

Citations in the corpus

Fast Inference of Mixture-of-Experts Language Models with Offloading (2023, neutral): “Most recently, there has been several concurrent works for quantizing Mixture-of-Experts models (Kim et al., 2023; Frantar & Alistarh, 2023b).”
QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models (2023, methodology): “Kim et al. (2022a) achieve 2-bit quantization on a 5 billion parameter MoE, which is considered relatively small in this area, by further optimization of the model via Quantization-Aware Training (Nagel et al., 2021).”