FDML: A Collaborative Machine Learning Framework for Distributed Features

By Yaochen Hu, Di Niu, Jianming Yang, Shengping Zhou. 2019

196 citations recorded by the source corpus.

Most current distributed machine learning systems try to scale up model training by using a data-parallel architecture that divides the computation for different samples among workers. We study distributed machine learning from a different motivation, where the information about the same samples, e.g., users and objects, are owned by several parities that wish to collaborate but do not want to share raw data with each other. We propose an asynchronous stochastic gradient descent (SGD) algorithm for such a feature distributed machine learning (FDML) problem, to jointly learn from distributed features, with theoretical convergence guarantees under bounded asynchrony. Our algorithm does not require sharing the original features or even local model parameters between parties, thus preserving the data locality. The system can also easily incorporate differential privacy mechanisms to preserve a higher level of privacy. We implement the FDML system in a parameter server architecture and compare our system with fully centralized learning (which violates data locality) and learning based on only local features, through extensive experiments performed on both a public data set a9a, and a large dataset of 5,000,000 records and 8700 decentralized features from three collaborating apps at Tencent including Tencent MyApp, Tecent QQ Browser and Tencent Mobile Safeguard. Experimental results have demonstrated that the proposed FDML system can be used to significantly enhance app recommendation in Tencent MyApp by leveraging user and item features from other apps, while preserving the locality and privacy of features in each individual app to a high degree.

Subfields: Artificial Intelligence, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yang Liu
2. Bingsheng He
3. Qinbin Li
4. Qiang Yang
5. Tianjian Chen
6. Zeyi Wen
7. Zhaomin Wu
8. Yan Kang

In Computer Science Applications

1. Bingsheng He
2. Qiang Yang
3. Yang Liu
4. Qinbin Li

Citations in the corpus

• Backdoor attacks and defenses in feature-partitioned collaborative learning (2020, background): “…collaborative learning frameworks have been developed for models including trees (Cheng et al., 2019), linear and logistic regression (Yang et al., 2019b; Hardy et al., 2017; Gratton et al., 2018; Kikuchi et al., 2017; Hu et al., 2019a), and neural networks (Liu et al., 2018; Hu et al., 2019b).”
• Learning Privately over Distributed Features: An ADMM Sharing Approach (2019, background): “A few recent studies [8, 9, 10, 11, 12, 13, 14] have revisited vertically partitioned features under the setting of distributed learning, which is motivated by the increase in feature dimensionality as well as the opportunity of cooperation between multiple parties that may hold different aspects of”
• Vertical Federated Learning: Concepts, Advances, and Challenges (2022, methodology): “[27] proposed FDML, allowing each party to update its local model asynchronously but based on the same sequence of randomly sampled training data.”
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications (2021, methodology): “Therefore, the data features and the associated model parametric blocks are split among edge devices, for which the asynchronous SGD method can be applied for vertical FL [157].”
• FedBCD: A Communication-Efficient Collaborative Learning Framework for Distributed Features (2022, background): “Feature-partitionedcollaborativelearningproblemshave been studied in the setting of both DL [4], [5], [6] and FL [7], [8], [9], [10].”
• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection (2019, background): “[71] propose FDML for vertical FL assuming all parties have the labels.”
• A Communication Efficient Collaborative Learning Framework for Distributed Features (2019, methodology): “ransmitted data [4, 10, 9], adding expensive communication overhead to the architectures. Differential Privacy (DP) is also a commonly-adopted approach, but such approaches suffer from precision loss [6, 7]. In sample-partitioned FL[3], it is demonstrated experimentally that multiple local updates c”