Learning Privately over Distributed Features: An ADMM Sharing Approach

By Yaochen Hu, Peng Liu, Linglong Kong, Di Niu. 2019

39 citations recorded by the source corpus.

Distributed machine learning has been widely studied in order to handle exploding amount of data. In this paper, we study an important yet less visited distributed learning problem where features are inherently distributed or vertically partitioned among multiple parties, and sharing of raw data or model parameters among parties is prohibited due to privacy concerns. We propose an ADMM sharing framework to approach risk minimization over distributed features, where each party only needs to share a single value for each sample in the training process, thus minimizing the data leakage risk. We establish convergence and iteration complexity results for the proposed parallel ADMM algorithm under non-convex loss. We further introduce a novel differentially private ADMM sharing algorithm and bound the privacy guarantee with carefully designed noise perturbation. The experiments based on a prototype system shows that the proposed ADMM algorithms converge efficiently in a robust fashion, demonstrating advantage over gradient based methods especially for data set with high dimensional feature spaces.

Subfields: Artificial Intelligence, Federated & Distributed Learning, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yang Liu
2. Qiang Yang
3. Tianjian Chen
4. Yan Kang
5. Yong Cheng
6. Xinwei Zhang
7. Xinle Liang
8. Mingyi Hong

In Federated & Distributed Learning

1. Yang Liu
2. Qiang Yang
3. Tianjian Chen
4. Yan Kang
5. Xinwei Zhang
6. Mingyi Hong
7. Liping Li
8. Jiahuan Luo

In Privacy & Security

1. Yang Liu
2. Yan Kang
3. Qiang Yang
4. Tianjian Chen
5. Yong Cheng
6. Mingyi Hong
7. Liping Li
8. Jiahuan Luo

Citations in the corpus

• FedBCD: A Communication-Efficient Collaborative Learning Framework for Distributed Features (2022, background): “Feature-partitionedcollaborativelearningproblemshave been studied in the setting of both DL [4], [5], [6] and FL [7], [8], [9], [10].”
• Self-supervised Cross-silo Federated Neural Architecture Search (2021, background): “[1], [8]; 2) Vertical Federated Learning (VFL): VFL describes the scenarios where different organizations with a common set of users train a cooperative model to better utilize the data with distributed features [2], [9]–[12]; 3) Federated Transfer Learning (FTL): FTL applies to the cases where the ”
• A Communication Efficient Collaborative Learning Framework for Distributed Features (2019, background): “ransmitted data [4, 10, 9], adding expensive communication overhead to the architectures. Differential Privacy (DP) is also a commonly-adopted approach, but such approaches suffer from precision loss [6, 7]. In sample-partitioned FL[3], it is demonstrated experimentally that multiple local updates c”