FetchSGD: Communication-Efficient Federated Learning with Sketching

By Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph E. Gonzalez, Raman Arora. 2020

432 citations recorded by the source corpus.

Existing approaches to federated learning suffer from a communication bottleneck as well as convergence issues due to sparse client participation. In this paper we introduce a novel algorithm, called FetchSGD, to overcome these challenges. FetchSGD compresses model updates using a Count Sketch, and then takes advantage of the mergeability of sketches to combine model updates from many workers. A key insight in the design of FetchSGD is that, because the Count Sketch is linear, momentum and error accumulation can both be carried out within the sketch. This allows the algorithm to move momentum and error accumulation from clients to the central aggregator, overcoming the challenges of sparse client participation while still achieving high compression rates and good convergence. We prove that FetchSGD has favorable convergence guarantees, and we demonstrate its empirical effectiveness by training two residual networks and a transformer model.

Subfields: Artificial Intelligence, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

Citations in the corpus

FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “Following [140], we use PersonaChat, a chit-chat dataset consisting of conversations between Amazon Mechanical Turk workers, as the benchmark dataset for the large-scale natural language processing task.”
FedSpeech: Federated Text-to-Speech with Continual Learning (2021, background): “Typical communication architectures used in federated learning [ McMahan et al. , 2017; Rothchild et al. , 2020 ] aggregate the information (e.g., gradients or model parameters) and train a global model.”
Distributed Learning in Wireless Networks: Recent Progress and Future Challenges (2021, methodology): “In [56], the count sketch is used to compress the local ML parameters thus overcoming the challenges of sparse device participation while still achieving high compression rates and convergence speed.”
FedScale: Benchmarking Model and System Performance of Federated Learning (2021, background): “Instead of training a single global model, some efforts resort to training a mixture of models [10, 12], clustering clients over training [16], or enforcing guided client selection [24]; To tackle the scarce and heterogeneous device resource, FedAvg [29] reduces communication cost by performing mult”
Self-supervised Cross-silo Federated Neural Architecture Search (2021, background): “[1], [8]; 2) Vertical Federated Learning (VFL): VFL describes the scenarios where different organizations with a common set of users train a cooperative model to better utilize the data with distributed features [2], [9]–[12]; 3) Federated Transfer Learning (FTL): FTL applies to the cases where the…”
Federated Learning for Internet of Things: A Comprehensive Survey (2021, methodology): “Moreover, a new optimization algorithm is proposed in [195] for FL-based IoT networks, called FetchSGD, that can train high-quality models for communication efficiency.”
A Comprehensive Survey of Privacy-preserving Federated Learning (2021, background): “FL is a machine learning technique, that aims at collaboratively training a global machine learning model on multiple datasets distributed over separate clients/nodes without explicitly exchanging data samples between the clients [121, 150].”
Federated Learning Meets Blockchain in Edge Computing: Opportunities and Challenges (2021, methodology): “Moreover, a new optimization algorithm is proposed in [86] for FL-based IoT networks, called FetchSGD, that can train high-quality models for communication efficiency.”