Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning

By Jinhyun So, Basak Guler, Amir Salman Avestimehr. 2020

387 citations recorded by the source corpus.

Federated learning is a distributed framework for training machine learning models over the data residing at mobile devices, while protecting the privacy of individual users. A major bottleneck in scaling federated learning to a large number of users is the overhead of secure model aggregation across many users. In particular, the overhead of the state-of-the-art protocols for secure model aggregation grows quadratically with the number of users. In this article, we propose the first secure aggregation framework, named Turbo-Aggregate, that in a network with <inline-formula> <tex-math notation="LaTeX">$N$ </tex-math></inline-formula> users achieves a secure aggregation overhead of <inline-formula> <tex-math notation="LaTeX">$O(N\log {N})$ </tex-math></inline-formula>, as opposed to <inline-formula> <tex-math notation="LaTeX">$O(N^{2})$ </tex-math></inline-formula>, while tolerating up to a user dropout rate of 50%. Turbo-Aggregate employs a multi-group circular strategy for efficient model aggregation, and leverages additive secret sharing and novel coding techniques for injecting aggregation redundancy in order to handle user dropouts while guaranteeing user privacy. We experimentally demonstrate that Turbo-Aggregate achieves a total running time that grows almost linear in the number of users, and provides up to <inline-formula> <tex-math notation="LaTeX">$40\times $ </tex-math></inline-formula> speedup over the state-of-the-art protocols with up to <inline-formula> <tex-math notation="LaTeX">$N=200$ </tex-math></inline-formula> users. Our experiments also demonstrate the impact of model size and bandwidth on the performance of Turbo-Aggregate.

Subfields: Artificial Intelligence, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Federated & Distributed Learning

Citations in the corpus

FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “Speciﬁcally, to accomplish the aforementioned goals of achieving security, privacy, and robustness for FL, various cryptography and coding-theoretic approaches have been proposed to manipulate intermediate model data (see [113, 114]).”
Federated Learning with Buffered Asynchronous Aggregation (2021, background): “SecAgg is a privacy enhancing technology based on cryptographic primitives (Bonawitz et al., 2016; Bell et al., 2020; So et al., 2021) or hardware-based Trusted Execution Environment (TEE) (Karl et al.”
Papaya: Practical, Private, and Scalable Federated Learning (2021, background): “Existing SMPC-based SecAgg approaches (Bonawitz et al., 2016; Bell et al., 2020; So et al., 2021b) hinder asynchronous training, as they require cohort formulation and inter-client communication in each round.”
Learning from History for Byzantine Robust Optimization (2020, background): “Further, combining robustness with notions of privacy and security is also a crucial and challenging problem (He et al., 2020b; So et al., 2020a;b; Jin et al., 2020).”
Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge (2020, background): “Others [29, 30, 31, 32, 33, 34, 35, 36, 37] optimize the communication cost without considering edge computational limitations.”
Biscotti: A Blockchain System for Private and Secure Federated Learning (2021, background): “…have individually been defended in prior work through centralized anomaly detection [21], differential privacy [22], [23], [24], [25] or secure aggregation [26], [27], but to the best of our knowledge, a private and decentralized solution that solves both threats concurrently does not yet exist.”
Federated Machine Learning: Survey, Multi-Level Classification, Desirable Criteria and Future Directions in Communication and Networking Systems (2021, methodology): “• Turbo-Aggregate: Turbo-Aggregate [23] is a communication and security-oriented aggregation technique that is based on a multi-group circular strategy for aggregating the model updates.”
Secure Aggregation with Heterogeneous Quantization in Federated Learning (2020, background): “Second, local models can be securely aggregated at the central server to update the global model [1], [13]–[18].”
A Scalable Approach for Privacy-Preserving Collaborative Machine Learning (2020, methodology): “Our encoding strategy utilizes Lagrange coding from [40]2, which has been applied to other problems such as privacy-preserving offloading of a training task [36] and secure federated learning [35].”