Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption

By Stephen Hardy, Wilko Henecka, Hamish Ivey-Law, Richard Nock, Giorgio Patrini, Guillaume Smith, Brian Thorne. 2017

622 citations recorded by the source corpus.

Consider two data providers, each maintaining private records of different feature sets about common entities. They aim to learn a linear model jointly in a federated setting, namely, data is local and a shared model is trained from locally computed updates. In contrast with most work on distributed learning, in this scenario (i) data is split vertically, i.e. by features, (ii) only one data provider knows the target variable and (iii) entities are not linked across the data providers. Hence, to the challenge of private learning, we add the potentially negative consequences of mistakes in entity resolution. Our contribution is twofold. First, we describe a three-party end-to-end solution in two phases ---privacy-preserving entity resolution and federated logistic regression over messages encrypted with an additively homomorphic scheme---, secure against a honest-but-curious adversary. The system allows learning without either exposing data in the clear or sharing which entities the data providers have in common. Our implementation is as accurate as a naive non-private solution that brings all data in one place, and scales to problems with millions of entities with hundreds of features. Second, we provide what is to our knowledge the first formal analysis of the impact of entity resolution's mistakes on learning, with results on how optimal classifiers, empirical losses, margins and generalisation abilities are affected. Our results bring a clear and strong support for federated learning: under reasonable assumptions on the number and magnitude of entity resolution's mistakes, it can be extremely beneficial to carry out federated learning in the setting where each peer's data provides a significant uplift to the other.

Subfields: Artificial Intelligence, Federated & Distributed Learning, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Federated & Distributed Learning

In Privacy & Security

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “Independently of the eventual technical levy of carrying out record linkage privately [348], the task itself happens to be substantially noise prone in the real world [347] and only sparse results have addressed its impact on training models [198].”
FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “As demonstrated in Figure 3, FL has various topology deﬁnitions, such as vertical FL [34, 35, 36, 37, 38, 39, 40], split learning [41, 42], decentralized FL [43, 44, 45, 46], and Hierarchical FL [47, 48, 49, 50, 51, 52].”
Backdoor attacks and defenses in feature-partitioned collaborative learning (2020, background): “, 2019), linear and logistic regression (Yang et al., 2019b; Hardy et al., 2017; Gratton et al., 2018; Kikuchi et al., 2017; Hu et al., 2019a), and neural networks (Liu et al.”
Federated Learning and Differential Privacy: Software tools analysis, the Sherpa.ai FL framework and methodological guidelines for preserving data privacy (2020, neutral): “ation of local models’ parameters, and(3)the direct applicability of DP techniques for additional privacy. Among such machine learning models we cite neural networks [31], linear [32,33] and logistic [34] regression.2 5.3. Models deviating from the federated machine learning paradigm It is worth men”
A Communication Efficient Vertical Federated Learning Framework (2019, background): “Implementation with HE We investigate the efﬁciency of FedBCD-p running on an industrial VFL platform, FATE 1 , with homomorphic encryption (HE) applied using the Credit-FTL task.”
Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data (2019, background): “Although it has been shown that in adversarial settings information about the training data can still be inferred from these updates [11], additional mechanisms, such as homomorphic encryption of the updates [12], [13] or differentially private training [14], can be applied to fully conceal any info”
Federated Machine Learning (2019, methodology): “ly partitioned data, including Cooperative Statistical Analysis [15], association rule mining [65], secure linear regression [22, 32, 55], classification [16] and gradient descent [68]. Recently, Ref [27, 49] proposed a vertical federated learning scheme to train a privacy-preserving logistic regres”
SecureBoost: A Lossless Federated Learning Framework (2019, background): “In the past, several attempts have been made to address the user-privacy problem while exchanging data (Hardy et al. 2017; Mohassel and Zhang 2017).”
Split learning for health: Distributed deep learning without sharing raw patient data (2018, background): “That said, with regards to non-neural network based federated learning techniques, the work in [27] shows their applicability to vertically partitioned distributed data [33, 34, 26, 35] shows applicability to multi-task learning in distributed settings.”
Entity Resolution and Federated Learning get a Federated Resolution (2018, background): “At this moderate price, the main message that comes from our results is very general as it roughly states that "any entity resolution algorithm making errors bounded in size and magnitude, used before minimizing any sufficiently regularized loss, yields a minimizer classifier close to the optimum le”
A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection (2019, methodology): “One can adopt differential privacy [38], [56], [120] to inject random noises into the parameters or use SMC [20], [21], [66] to”
Federated Learning for Healthcare Informatics (2019, background): “Due to its success in cloud computing, it comes naturally into this realm, and it has certainly been used in many federated learning researches [14, 40].”
Vertical Federated Learning: Concepts, Advances, and Challenges (2022, methodology): “Label inference attacks using batch-level gradients: When the VFL applies the P-2 protocol, no intermediate result exchanged among parties is revealed to any party (e.g., encrypted by HE [97]).”
Parallel Distributed Logistic Regression for Vertical Federated Learning without Third-Party Coordinator (2019, background): “As compared with [Hardy et al., 2017], our solution is different in the following three perspectives.”
Federated Machine Learning: Survey, Multi-Level Classification, Desirable Criteria and Future Directions in Communication and Networking Systems (2021, background): “For example, the authors of [163] address the challenge of private federated learning wherein (1) data are vertically divided across clients by features; (2) target variable is known by only one party; and (3) entities’ data are not linked across clients.”
A Communication Efficient Collaborative Learning Framework for Distributed Features (2019, background): “to consider cross-organizational collaborative learning problems where parties share the same users with different set of features, and this scenario is classiﬁed as vertical federated learning (VFL) [2, 3]. Existing architectures for VFL still face several critical challenges and communication over”
HybridAlpha: An Efficient Approach for Privacy-Preserving Federated Learning (2019, methodology): “To tackle the issues and challenges in the case of vertically partitioned data, several methods are proposed in [9, 14, 22], which focus on entity resolution and simple machine learning models, like logistic regression.”
Trustworthy AI: From Principles to Practices (2021, methodology): “Reference [148] used homomorphic encryption for vertical logistic regression-based model learning.”