On Large-Cohort Training for Federated Learning

By Zachary B. Charles, Zachary Garrett, Zhouyuan Huo, Sergei Shmulyian, Virginia Smith. 2021

126 citations recorded by the source corpus.

Federated learning methods typically learn a model by iteratively sampling updates from a population of clients. In this work, we explore how the number of clients sampled at each round (the cohort size) impacts the quality of the learned model and the training dynamics of federated learning algorithms. Our work poses three fundamental questions. First, what challenges arise when trying to scale federated learning to larger cohorts? Second, what parallels exist between cohort sizes in federated learning and batch sizes in centralized learning? Last, how can we design federated learning methods that effectively utilize larger cohort sizes? We give partial answers to these questions based on extensive empirical evaluation. Our work highlights a number of challenges stemming from the use of larger cohorts. While some of these (such as generalization issues and diminishing returns) are analogs of large-batch training challenges, others (including training failures and fairness concerns) are unique to federated learning.

Subfields: Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Federated & Distributed Learning

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “FL-1-CSH converges roughly 2× faster but results in higher energy consumption, as large cohort training is known to be less efficient (Charles et al., 2021).”
Federated Learning with Buffered Asynchronous Aggregation (2021, background): “To We assume the time a client requires to perform local training is proportional to the number of examples the client has, same as the assumption in Charles et al. (2021).”
Stochastic Controlled Averaging for Federated Learning with Communication Compression (2023, background): “This slows down the convergence of the global model because of less accessible data/information per round (Charles et al., 2021; Chen et al., 2022; Li and Li, 2023).”
Papaya: Practical, Private, and Scalable Federated Learning (2021, background): “Large-cohort training has been found to make inefﬁcient use of client updates (Bonawitz et al., 2019; Charles et al., 2021).”