SecureBoost: A Lossless Federated Learning Framework

By Kewei Cheng, Tao Fan, Yilun Jin, Yang Liu, Tianjian Chen, Qiang Yang. 2019

701 citations recorded by the source corpus.

The protection of user privacy is an important concern in machine learning, as evidenced by the rolling out of the General Data Protection Regulation (GDPR) in the European Union (EU) in May 2018. The GDPR is designed to give users more control over their personal data, which motivates us to explore machine learning frameworks for data sharing that do not violate user privacy. To meet this goal, in this article, we propose a novel lossless privacy-preserving tree-boosting system known as SecureBoost in the setting of federated learning. SecureBoost first conducts entity alignment under a privacy-preserving protocol and then constructs boosting trees across multiple parties with a carefully designed encryption strategy. This federated learning system allows the learning process to be jointly conducted over multiple parties with common user samples but different feature sets, which corresponds to a vertically partitioned dataset. An advantage of SecureBoost is that it provides the same level of accuracy as the non -privacy-preserving approach while at the same time, reveals no information of each private data provider. We show that the SecureBoost framework is as accurate as other nonfederated gradient tree-boosting algorithms that require centralized data, and thus, it is highly scalable and practical for industrial applications such as credit risk analysis. To this end, we discuss information leakage during the protocol execution and propose ways to provably reduce it.

Subfields: Artificial Intelligence, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Federated & Distributed Learning

Citations in the corpus

SplitNN-driven Vertical Partitioning (2020, background): “Vertically partitioned decision trees The works in (Gascón et al. , 2016; Vaidya et al. , 2008; Vaidya & Clifton, 2004; Kourtellis et al. , 2016; Cheng et al. , 2019) share approaches for vertically partitioned learning with decision trees, gradient boosted decision trees and Hoeffding trees.”
FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “…application domains include, but are not limited to, computer vision [77, 78], natural language processing [79, 80, 81, 82, 83], ﬁnance [74, 35, 84], transportation [85, 86, 87, 88, 89, 90, 91, 92, 93, 94], digital health [95, 96, 97, 98, 99, 100, 101], recommendation [102, 103, 104, 105,…”
Real-World Image Datasets for Federated Learning (2019, methodology): “So far, federated learning has been implemented with many state-of-the-art algorithms such as long short-term memory (LSTM) networks [McMahan et al. , 2017], support vector machines (SVMs) [Rubinstein et al. , 2009], gradient boosting trees [Cheng et al. , 2019], etc..”
A Survey on Federated Learning for Resource-Constrained IoT Devices (2021, positive): “In tree-based FL, a model is generated for training single or multiple DTs. Cheng et al. [25] and Li et al. [30] designed gradient boosting DTs (GBDTs) by considering both horizontal and vertical partitioned data schemes.”
FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare (2019, methodology): “It is important to note that FedHealth does not perform parameter sharing as in [Cheng et al., 2019] for computational efficiency.”
Privacy Preserving Machine Learning with Homomorphic Encryption and Federated Learning (2021, methodology): “About performance, in [37], it proposed a framework SecureBoost with almost the same accuracy compared with the ﬁve privacy protection methods.”
BatchCrypt: Efficient Homomorphic Encryption for Cross-Silo Federated Learning (2020, methodology): “Many recent works [13, 37, 38,48] advocate the use of additively HE schemes, notably Paillier [46], as the primary means of privacy guarantee in cross-silo FL: each client transfers the encrypted local updates to the server for direct aggregation; the result is then sent back to each client for…”
Federated Learning for Healthcare Informatics (2019, background): “Thus, some researchers combine DP with SMC to reduce the growth of noise injection as the number of parties increases without sacrificing privacy while preserving provable privacy guarantees, protecting against extraction attacks and collusion threats [18, 100].”
A Quasi-Newton Method Based Vertical Federated Learning Framework for Logistic Regression (2019, background): “In particular, [7] proposes a privacy-preserving tree-boosting system SecureBoost and [8] propose a logistic regression framework for vertically partitioned data.”
Cloud-based Federated Boosting for Mobile Crowdsensing (2020, methodology): “Recent studies pointed out that the federated learning architectures are vulnerable to user data reconstruction attack [27],[6].”
HybridAlpha: An Efficient Approach for Privacy-Preserving Federated Learning (2019, methodology): “To tackle the issues and challenges in the case of vertically partitioned data, several methods are proposed in [9, 14, 22], which focus on entity resolution and simple machine learning models, like logistic regression.”
A Comprehensive Survey of Privacy-preserving Federated Learning (2021, methodology): “[27] proposed a lossless vertical FL method, which could enable clients to train gradient boosting decision trees in a collaborative manner.”
Faster On-Device Training Using New Federated Momentum Algorithm (2020, methodology): “It has been widely applied for mobile keyboard prediction [9], private language models [25], and financial-client classification [7].”
FATE: An Industrial Grade Platform for Collaborative Learning With Data Protection (2021, methodology): “Additional performance benchmarks for implementing advanced algorithms on FATE include XGBoost (Cheng et al., 2021) and FTL (Liu et al.”
A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection (2019, methodology): “If the parties have the same sample space but different features, one can use FedBCD [111] for NNs and SecureBoost [36] for trees.”
A Survey on Federated Learning: The Journey From Centralized to Distributed On-Site Learning and Beyond (2021, methodology): “SecureBoost, a lossless tree-boosting system for privacy preserving, is proposed in [106] using HE.”