Agnostic Federated Learning
By Mehryar Mohri, Gary Sivek, Ananda Theertha Suresh. 2019
1,132 citations recorded by the source corpus.
A key learning scenario in large-scale applications is that of federated learning, where a centralized model is trained based on data originating from a large number of clients. We argue that, with the existing training and inference, federated models can be biased towards different clients. Instead, we propose a new framework of agnostic federated learning, where the centralized model is optimized for any target distribution formed by a mixture of the client distributions. We further show that this framework naturally yields a notion of fairness. We present data-dependent Rademacher complexity guarantees for learning with this objective, which guide the definition of an algorithm for agnostic federated learning. We also give a fast stochastic optimization algorithm for solving the corresponding optimization problem, for which we prove convergence bounds, assuming a convex loss function and hypothesis set. We further empirically demonstrate the benefits of our approach in several datasets. Beyond federated learning, our framework and algorithm can be of interest to other learning scenarios such as cloud computing, domain adaptation, drifting, and other contexts where the training and test distributions do not coincide.
Subfields: Artificial Intelligence, Federated & Distributed Learning, Statistical Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Martin Jaggi
- Qiang Yang
- Dacheng Tao
- Sergey Levine
- Yoshua Bengio
- Quoc V. Le
- Ruslan Salakhutdinov
In Federated & Distributed Learning
In Statistical Learning Theory
- Sergey Levine
- Alekh Agarwal
- Ananda Theertha Suresh
- Jakub Konecný
- Mehryar Mohri
- Virginia Smith
- Sanjiv Kumar
- Tian Li
Citations in the corpus
- Advances and Open Problems in Federated Learning (2019, background): “Can we guarantee that the model hFL learned via federated learning is at least as accurate as hk when used for client k? Can we quantify how much improvement can be expected via federated leaning? And can we develop personalization strategies with theoretical guarantees that at least match the perfo”
- Ensemble Distillation for Robust Model Fusion in Federated Learning (2020, background): “, 2016) and its recent adaptations (Mohri et al., 2019; Li et al., 2020; Hsu et al., 2019; Karimireddy et al., 2019; Hsu et al., 2020; Reddi et al., 2020), are all based on directly averaging of the participating client’s parameters and can hence only be applied if all client’s models have the same ”
- Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning (2020, background): “[24] Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh.”
- FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “ICML 2019 Agnostic Federated Learning [147] UCI Adult Census datase logistic regression 10 Fashion-MNIST logistic regression 10 Cornell movie dataset two-layer LSTM mode 10 Penn TreeBank (PTB) dataset two-layer LSTM mode 10 ICML 2019 Bayesian Nonparametric Federated Learning of Neural Networks [145]”
- Byzantine-Resilient Secure Federated Learning (2020, background): “Another important direction in federated learning is the study of fairness and how to avoid biasing the model towards specific users [28], [29].”
- A Theory of Multiple-Source Adaptation with Limited Target Labeled Data (2020, methodology): “However, one can use the min-max mirror descent algorithm or stochastic minmax mirror descent algorithms (Juditsky et al., 2011; Namkoong and Duchi, 2016; Cotter et al., 2019; Mohri et al., 2019) to obtain heuristic solutions.”
- Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization (2020, methodology): “Moreover, FedNova is also compatible with and complementary to gradient compression/quantization [43–48] and fair aggregation techniques [49, 50].”
- On the Outsized Importance of Learning Rates in Local Update Methods (2020, background): “…work on the intersection of federated learning with topics including differential privacy (McMahan et al., 2018; Augenstein et al., 2020), fairness (Mohri et al., 2019; Li et al., 2020b), robustness (Ghosh et al., 2019; Bagdasaryan et al., 2018; Sun et al., 2019), and communication-efficiency…”
- Three Approaches for Personalization with Applications to Federated Learning (2020, methodology): “This can be trained using either standard empirical risk minimization (Vapnik, 1992) or other methods such as agnostic risk minimization (Mohri et al., 2019).”
- Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning (2020, background): “References [42], [43] have explored federated learning frameworks to address fairness challenges and to avoid biasing the trained model towards certain users.”
- Robust Aggregation for Federated Learning (2019, background): “Extensions were proposed in [7, 29, 33, 47, 50, 57, 69, 75, 79]; see also the recent surveys [46, 56].”
- Secure Federated Submodel Learning (2019, background): “[36] considered an unfairness issue that the global model can be unevenly biased toward different clients.”
- SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning (2019, methodology): “…several types of issues and has been the topic of multiple research efforts studying the learning and generalization properties in that scenario (Mohri et al., 2019), systems, networking and communication bottleneck problems due to frequent exchanges between the central server and the clients,…”
- Differential Privacy Has Disparate Impact on Model Accuracy (2019, background): “Prior work on subgroup fairness aims to achieve good accuracy on all subgroups [21] using agnostic learning [22, 29].”
- Fair Resource Allocation in Federated Learning (2019, methodology): “Recent work that has been proposed specifically for the federated setting has also only been applied at small scales (2-3 groups/devices), and lacks flexibility by optimizing only the performance of the single worst device (Mohri et al., 2019).”
- SecureBoost: A Lossless Federated Learning Framework (2019, background): “Following the same idea, several attempts have been made to reinvent different machine learning models to the federated setting, including decision tree [20], [21], linear/logistic regression [22], [23], [24] and neural network [25], [26].”
- Federated Learning: Challenges, Methods, and Future Directions (2019, methodology): “Recent works have proposed modified modeling approaches that aim to reduce the variance of the model performance across devices [19], [26], [33].”
- Adaptive Personalized Federated Learning (2020, methodology): “…updates the local version of the global model using its own local data, (iii) the server aggregates over the obtained local models to update the global model, and this process continues until convergence (McMahan et al., 2017; Mohri et al., 2019; Karimireddy et al., 2019; Pillutla et al., 2019).”
- Federated Learning with Matched Averaging (2020, background): “Agnostic Federated Learning (AFL) (Mohri et al., 2019), as another variant of FedAvg, optimizes a centralized distribution that is a mixture of the client distributions.”