Personalized and Private Peer-to-Peer Machine Learning

By Aurélien Bellet, Rachid Guerraoui, Mahsa Taziki, Marc Tommasi. 2017

172 citations recorded by the source corpus.

The rise of connected personal devices together with privacy concerns call for machine learning algorithms capable of leveraging the data of a large number of agents to learn personalized models under strong privacy requirements. In this paper, we introduce an efficient algorithm to address the above problem in a fully decentralized (peer-to-peer) and asynchronous fashion, with provable convergence rate. We show how to make the algorithm differentially private to protect against the disclosure of information about the personal datasets, and formally analyze the trade-off between utility and privacy. Our experiments show that our approach dramatically outperforms previous work in the non-private case, and that under privacy constraints, we can significantly improve over models learned in isolation.

Subfields: Artificial Intelligence, Information Systems, Federated & Distributed Learning, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Federated & Distributed Learning

In Privacy & Security

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “The setting of decentralized SGD also naturally lends itself to asynchronous algorithms in which each client becomes active independently at random times, removing the need for global synchronization and potentially improving scalability [110, 392, 54, 30, 267].”
Privacy Amplification by Decentralization (2020, methodology): “[37, 18, 52, 38, 51, 11, 45, 35] for recent work and [32] (Section 2.”
Salvaging Federated Learning by Local Adaptation (2020, background): “Peer-topeer (not federated) learning with convex losses and without robustness is studied in (Bellet et al., 2018).”
Central Server Free Federated Learning over Single-sided Trust Social Networks (2019, background): “…the data privacy and avoid the communication bottleneck, the decentralized architecture has been recently proposed (Vanhaesebrouck et al., 2017; Bellet et al., 2018), where the centralized node has been removed, and each node only communicates with its neighbors (with mutual trust) by…”
Fully Decentralized Joint Learning of Personalized Models and Collaboration Graphs (2019, methodology): “As our algorithm communicates very scarcely, we think that the privacy/accuracy trade-off may be better than the one known for linear models (Bellet et al., 2018).”
Decentralized Federated Learning: A Survey and Perspective (2023, background): “Bellet et al. [149] introduced an asynchronous and differential privacy algorithm in DFL to safeguard user privacy.”
Personalized Federated Learning: An Attentive Collaboration Approach (2020, background): “Distributed multi-task learning [33, 34, 35, 36] tried to conduct multi-task learning when the data for each task is separately owned by different clients.”
Throughput-Optimal Topology Design for Cross-Silo Federated Learning (2020, background): “A current trend [104, 18, 100, 95, 7, 49, 53] is then to replace communication with the orchestrator by peer-to-peer communications between individual silos, which perform local partial aggregations of model updates.”
Personalized Federated Learning for Intelligent IoT Applications: A Cloud-Edge Based Framework (2020, background): “, IoT based personalized smart healthcare, smart home services and applications, fine-grained location-aware recommendation services, and on-premise intelligent video analytics), personalized learning has recently attracted great attention [5], [6].”
DP-ADMM: ADMM-Based Distributed Learning With Differential Privacy (2018, background): “Among the works on distributed learning with differential privacy, most of them focus on subgradient-based algorithms [35]–[38] and only a few works consider ADMMbased methods [1], [2], [17]–[19].”
Federated Multi-Task Learning under a Mixture of Distributions (2021, background): “For these reasons, they have attracted significant interest recently in the machine learning community [41, 63, 42, 62, 3, 51, 46, 31].”
Who started this rumor? Quantifying the natural differential privacy guarantees of gossip protocols (2019, unknown): “Showing that gossip protocols can naturally amplify diﬀerential privacy for the message contents would make them very desirable for privacy-preserving distributed AI applications, such as those based on federated [23] and decentralized machine learning [42].”
Towards Decentralized Deep Learning with Differential Privacy (2019, methodology): “For distributed machine learning, Bellet et al. [8] proposed a completely decentralized and asynchronous algorithm to solve personalized optimization problem and also use DP in their privacy-preserving method.”