ZipML: Training Linear Models with End-to-End Low Precision, and a Little Bit of Deep Learning

By Hantian Zhang, Jerry Zheng Li, Kaan Kara, Dan Alistarh, Ji Liu, Ce Zhang. 2017

213 citations recorded by the source corpus.

Subfields: Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, background): “In [4, 40, 44], random quantization schemes are studied.”
Communication Efficient Decentralized Training with Multiple Local Updates (2019, background): “The more direct approach is to reduce the size of the messages through gradient compression or sparsiﬁcation [32, 23, 51, 40, 42, 8].”
$\texttt{DeepSqueeze}$: Decentralization Meets Error-Compensated Compression (2019, background): “Previous works mostly focus on a centralized scenario where the compression is assumed to be unbiased [Jiang and Agrawal, 2018, Shen et al., 2018, Wangni et al., 2018, Wen et al., 2017, Zhang et al., 2017].”
Natural Compression for Distributed Deep Learning (2019, background): “[49] Hantian Zhang, Jerry Li, Kaan Kara, Dan Alistarh, Ji Liu, and Ce Zhang.”
Communication trade-offs for synchronized distributed SGD with large step size (2019, background): “Some of the work in this direction can be found in Zhang et al. (2017); Wen et al. (2017); Wangni et al. (2017); Sa et al. (2015); Na et al. (2017); Gupta et al. (2015); Alistarh et al. (2016); Khirirat et al. (2018).”
Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication (2019, background): “A class of very common quantization operators is based on random dithering (Goodall, 1951; Roberts, 1962) that is in addition also unbiased, E ξ Q ( x ) = x , ∀ x ∈ R d , see (Alistarh et al., 2017; Wen et al., 2017; Zhang et al., 2017).”
Federated Learning: Challenges, Methods, and Future Directions (2019, background): “ed at each round. These methods have been extensively studied, both empirically and theoretically, in previous literature for distributed training in data center environments; we defer the readers to [119, 135] for a more complete review. In federated environments, the low participation of devices, ”
Model compression via distillation and quantization (2018, methodology): “ea of optimizing the locations of quantization points during the learning process, which we use in differentiable quantization, has been used previously in Lan et al. (2014); Koren & Sill (2011); Zhang et al. (2017), although in the different context of matrix completion and recommender systems.”
Sparsified SGD with Memory (2018, methodology): “The ZipML framework also applies this technique to the data [44].”
Communication Compression for Decentralized Training (2018, methodology): “The commonly used stochastic unbiased compression include random quantization 1 [Zhang et al., 2017a] and sparsiﬁcation 2 [Wangni et al., 2017, Konečn`y and Richtárik, 2016].”
Gradient Sparsification for Communication-Efficient Distributed Optimization (2017, background): “Several papers considered the problem of reducing the precision of gradient by using fewer bits to represent floating-point numbers [25, 2, 34, 8, 32] or only transmitting coordinates of large magnitudes[1, 21].”
ATOMO: Communication-efficient Learning via Atomic Sparsification (2018, neutral): “Partially initiated by a 1-bit implementation of SGD by Microsoft in [5], a large number of recent studies revisited the idea of low-precision training as a means to reduce communication [13, 14, 15, 16, 17, 18, 19, 17, 20, 21].”
Stochastic Distributed Learning with Gradient Quantization and Variance Reduction (2019, background): “…is much slower than computation and the communication bottleneck between workers is a major limiting factor for many large scale applications (such as e.g. deep neural network training), as reported in e.g. (Seide et al., 2014; Alistarh et al., 2017; Zhang et al., 2017; Lin et al., 2018).”
High-Dimensional Stochastic Gradient Quantization for Communication-Efficient Edge Learning (2020, background): “To tackle this challenge, numerous schemes have been developed for compressing stochastic gradients to reduce the said communication overhead [6], [7].”
Training DNNs with Hybrid Block Floating Point (2018, background): “ZipML [15], DoReFa [6], and Flexpoint [5] train DNNs with end-to-end low precision.”
High-Dimensional Stochastic Gradient Quantization for Communication-Efficient Edge Learning (2019, background): “of parameters), one main theme in the federated learning research is to develop communication-efficient strategies for fast update-uploading to accelerate learning [3]–[5].”