The ZipML Framework for Training Models with End-to-End Low Precision: The Cans, the Cannots, and a Little Bit of Deep Learning

By Hantian Zhang, Jerry Zheng Li, Kaan Kara, Dan Alistarh, Ji Liu, Ce Zhang. 2016

20 citations recorded by the source corpus.

Recently there has been significant interest in training machine-learning models at low precision: by reducing precision, one can reduce computation and communication by one order of magnitude. We examine training at reduced precision, both from a theoretical and practical perspective, and ask: is it possible to train models at end-to-end low precision with provable guarantees? Can this lead to consistent order-of-magnitude speedups? We present a framework called ZipML to answer these questions. For linear models, the answer is yes. We develop a simple framework based on one simple but novel strategy called double sampling. Our framework is able to execute training at low precision with no bias, guaranteeing convergence, whereas naive quantization would introduce significant bias. We validate our framework across a range of applications, and show that it enables an FPGA prototype that is up to 6.5x faster than an implementation using full 32-bit precision. We further develop a variance-optimal stochastic quantization strategy and show that it can make a significant difference in a variety of settings. When applied to linear models together with double sampling, we save up to another 1.7x in data movement compared with uniform quantization. When training deep networks with quantized models, we achieve higher accuracy than the state-of-the-art XNOR-Net. Finally, we extend our framework through approximation to non-linear models, such as SVM. We show that, although using low-precision data induces bias, we can appropriately bound and control the bias. We find in practice 8-bit precision is often sufficient to converge to the correct solution. Interestingly, however, in practice we notice that our framework does not always outperform the naive rounding approach. We discuss this negative result in detail.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Communication trade-offs for synchronized distributed SGD with large step size (2019, background): “While OSA requires only a single communication step, it typically does not perform very well in practice (Zhang et al., 2016).”
High-Accuracy Low-Precision Training (2018, background): “Recently, Zhang et al. [32] has developed techniques called double sampling and optimal quantization which enable users to quantize the training dataset with provable guarantees on the accuracy for convex linear models.”
ZipML: Training Linear Models with End-to-End Low Precision, and a Little Bit of Deep Learning (2017, methodology): “In the full version of this paper (Zhang et al., 2016), we present an algorithm which, given Ω and k, provides a split using at most 4k intervals, which guarantees a 2-approximation of the optimal variance for k intervals, using O(N logN) time.”
Communication trade-offs for Local-SGD with large step size (2019, background): “At the other extreme, MBA performs better in practice, but the number of communications equals the number of steps, which is a major burden, as communication is highly time consuming [22].”
CodedPrivateML: A Fast and Privacy-Preserving Framework for Distributed Machine Learning (2019, methodology): “1 in (Zhang et al., 2016) using the quantized weights formed in Section 3.”
Low-Memory Neural Network Training: A Technical Report (2019, methodology): “This has been studied by, for instance, Choi et al. (2019), Jain et al. (2018, 2019), Zhang et al. (2016).”