Sparse Binary Compression: Towards Distributed Deep Learning with minimal Communication

By Felix Sattler, Simon Wiedemann, K. Müller, Wojciech Samek. 2018

243 citations recorded by the source corpus.

Currently, progressively larger deep neural networks are trained on ever growing data corpora. In result, distributed training schemes are becoming increasingly relevant. A major issue in distributed training is the limited communication bandwidth between contributing nodes or prohibitive communication cost in general. To mitigate this problem we propose Sparse Binary Compression (SBC), a compression framework that allows for a drastic reduction of communication cost for distributed training. SBC combines existing techniques of communication delay and gradient sparsification with a novel binarization method and optimal weight update encoding to push compression gains to new limits. By doing so, our method also allows us to smoothly trade-off gradient sparsity and temporal sparsity to adapt to the requirements of the learning task. Our experiments show, that SBC can reduce the upstream communication on a variety of convolutional and recurrent neural network architectures by more than four orders of magnitude without significantly harming the convergence speed in terms of forward-backward passes. For instance, we can train ResNet50 on ImageNet in the same number of iterations to the baseline accuracy, using ×3531 less bits or train it to a 1% lower accuracy using ×37208 less bits. In the latter case, the total upstream communication required is cut from 125 terabytes to 3.35 gigabytes for every participating client.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling (2019, background): “There have been massive amount of work in the context of algorithms [37, 33, 36, 5, 26] and systems [16, 43, 9] that improve the communication efficiency of synchronous distributed SGD in a fully-connected network.”
Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data (2019, methodology): “Over the course of multiple hundred thousands of training iterations on big data sets, the total communication for every client can easily grow to more than a petabyte [17].”
Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints (2019, background): “Different authors have investigated the convergence properties of Federated Learning in congruent iid and non-iid scenarios: [12],[13],[14] and [15] perform an empirical investigation, [8], [16], [17] and [9] prove convergence guarantees.”
Entropy-Constrained Training of Deep Neural Networks (2018, unknown): “We set their respective initial cardinalities, K̄, as follows: (i) LeNet-300-100, K̄ = [3, 3, 33], (ii) LeNet-5, K̄ = [5, 5, 33], and (iii) VGG-Cifar-10, K̄ = [33, 17, 15, 11, 7, 7, 7, 5, 5, 5, 3, 3, 3, 17, 33].”
Compact and Computationally Efficient Representation of Deep Neural Networks (2018, background): “The combination of entropy-constrained regularization and quantization and entropy-optimized data formats may push the limits of neural network compression even further and also be beneficial for applications such as federated or distributed learning [56], [57].”
A Survey on Federated Learning: The Journey From Centralized to Distributed On-Site Learning and Beyond (2021, background): “results in communicating a full model in each round, which may reach a size of gigabytes in huge parameters-based models [50].”
Machine Learning in the Air (2019, background): “Sparsiﬁcation, on the other hand, reduces the communication load by transmitting only the important values of local estimates [72]–[74].”
Federated Learning Over Wireless Fading Channels (2019, methodology): “The scheduled device quantizes its gradient estimate to a finite number of bits using the gradient compression scheme in [18] while accumulating the error from previous iterations (this will be clarified later), and employs a channel code to transmit the bits over the available bandwidth-limited cha”
Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air (2019, methodology): “We observed empirically that this improves the performance compared to the standard approach in the literature, where the same compression scheme is applied at each iteration [28].”
Decentralized Federated Learning: A Survey and Perspective (2023, background): “Even though lightweight models are emerging recently [95], [96], the need to compute and store model data can easily reach petabytes in size [97].”
Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications (2022, unknown): “Then, each device, depending on the rate available to it, reduces the size of its model update by compressing it using the technique proposed in [265].”
Wireless Federated Distillation for Distributed Edge Learning with Heterogeneous Data (2019, methodology): “Accordingly, each device k at the i-th global iteration computes the vector v i = sparseq ( ∆w i + ∆ k i ) , where the accumulated quantization error is updated as [10], [11] ∆ i = ∆w k i + ∆ k i −Qb ( v i ) .”
Over-the-Air Machine Learning at the Wireless Edge (2019, methodology): “Numerous studies have focused on communication-efficient DSGD schemes, where three main approaches are employed; namely, quantization [1]–[3], sparsification [4], [6], [7], and local updates [8], [9].”
Communication-Efficient Federated Learning for Wireless Edge Intelligence in IoT (2020, unknown): “[12] demonstrated their sparse binary compression system against uncompressed FedAvg, and were able to”
One-Bit Over-the-Air Aggregation for Communication-Efficient Federated Edge Learning: Design and Convergence Analysis (2020, background): “The last approach focuses on update compression by exploiting the sparsity of gradient updates [12], [13] and low-resolution gradient/model-parameter quantization [14]–[16].”