Pieces of Eight: 8-bit Neural Machine Translation

By Jerry Quinn, Miguel Ballesteros. 2018

25 citations recorded by the source corpus.

Neural machine translation has achieved levels of fluency and adequacy that would have been surprising a short time ago. Output quality is extremely relevant for industry purposes, however it is equally important to produce results in the shortest time possible, mainly for latency-sensitive applications and to control cloud hosting costs. In this paper we show the effectiveness of translating with 8-bit quantization for models that have been trained using 32-bit floating point values. Results show that 8-bit translation makes a non-negligible impact in terms of speed with no degradation in accuracy and adequacy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

Scaling Neural Machine Translation (2018, background): “There has also been recent work on using lower precision for inference only (Quinn and Ballesteros, 2018).”
Towards Fully 8-bit Integer Inference for the Transformer Model (2020, methodology): “One appealing solution to these issues is to reduce the numerical precision used in the model at hand [Hubara et al. , 2016; Micikevicius et al. , 2018], where both the parameters and the activations are represented with fewer bits.”
Gender bias amplification during Speed-Quality optimization in Neural Machine Translation (2021, unknown): “Low-bit quantization (INT8) is another recent approach which improves decoding speed and reduces the memory footprint of models (Zafrir et al., 2019; Quinn and Ballesteros, 2018).”
Sharing Attention Weights for Fast Transformer (2019, background): “These include vocabulary selection [L’Hostis et al. , 2016; Sankaran et al. , 2017], knowledge distillation [Hinton et al. , 2015; Kim and Rush, 2016], low-precision computation [Micikevicius et al. , 2018; Quinn and Ballesteros, 2018], recurrent stacked networks [Dabre and Fujita, 2019] and etc.”
The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation (2021, methodology): “Quantization can be used to speed up inference and relax hardware requirements, as has been shown for e.g., 8-bit (Quinn and Ballesteros, 2018), 4-bit (Aji and Heaﬁeld, 2020) and recently also below 3-bit quantization (Chung et al., 2020) of NMT models.”