Natural Compression for Distributed Deep Learning

By Samuel Horváth, Chen-Yu Ho, Ludovit Horvath, Atal Narayan Sahu, Marco Canini, Peter Richtárik. 2019

183 citations recorded by the source corpus.

Modern deep learning models are often trained in parallel over a collection of distributed machines to reduce training time. In such settings, communication of model updates among machines becomes a significant performance bottleneck and various lossy update compression techniques have been proposed to alleviate this problem. In this work, we introduce a new, simple yet theoretically and practically effective compression technique: {\em natural compression (NC)}. Our technique is applied individually to all entries of the to-be-compressed update vector and works by randomized rounding to the nearest (negative or positive) power of two, which can be computed in a "natural" way by ignoring the mantissa. We show that compared to no compression, NC increases the second moment of the compressed vector by not more than the tiny factor $\nicefrac{9}{8}$, which means that the effect of NC on the convergence speed of popular training algorithms, such as distributed SGD, is negligible. However, the communications savings enabled by NC are substantial, leading to {\em $3$-$4\times$ improvement in overall theoretical running time}. For applications requiring more aggressive compression, we generalize NC to {\em natural dithering}, which we prove is {\em exponentially better} than the common random dithering technique. Our compression operators can be used on their own or in combination with existing operators for a more aggressive combined effect, and offer new state-of-the-art both in theory and practice.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Federated & Distributed Learning

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “Much of the existing literature applies to the objective (a) [245, 376, 244, 20, 204].”
Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization (2020, methodology): “The variance-reduced CGD method DIANA (Mishchenko et al., 2019a; Horv´ath et al., 2019b) enjoys even better behavior: the number of communication rounds for this method is unaﬀected up to a certain level of compression when the variance induced by compression reaches a certain threshold.”
Scaling Distributed Machine Learning with In-Network Aggregation (2019, unknown): “This innovation will continue (as work [40, 70] that builds upon our architecture demonstrates); our goal is not to propose new representations but to demonstrate that techniques like those in the literature are practical with programmable switches.”
Uncertainty Principle for Communication Compression in Distributed and Federated Learning and the Search for an Optimal Compressor (2020, unknown): “…top-k sparsiﬁcation (Alistarh et al., 2018), standard random dithering (Goodall, 1951; Roberts, 1962; Alistarh et al., 2017), natural dithering (Horv´ath et al., 2019a), ternary quantization (Wen et al., 2017), and scaled sign quantization (Karimireddy et al., 2019b; Bernstein et al., 2018,…”
Stochastic Controlled Averaging for Federated Learning with Communication Compression (2023, background): “Notable examples include Sign-SGD (Seide et al., 2014; Bernstein et al., 2018), low-bit fixed rounding (Dettmers, 2016), Q-SGD (Alistarh et al., 2017), Turn-Grad (Wen et al., 2017), and natural compression (Horv´oth et al., 2022).”
Artemis: tight convergence guarantees for bidirectional compression in Federated Learning (2020, background): “Combining Assumptions 3 and 4 leads to the following upper bound a.s: Finally, compression operators can be classiﬁed in three main categories: quantization [as in 30, 39, 3, 34], sparsiﬁcation [as in 2, 4, 17] and dithering [as in 14].”
Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning (2019, unknown): “Stochastic rounding is a well-known technique achieving unbiased quantization, but existing works [51], [55]–[58] focus on quantizing individual gradients, not the gradient differences.”
FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout (2021, methodology): “Building upon gradient quantisation methods [47, 26, 56, 28], [2] proposed using quantisation in the model sharing and aggregation steps.”