Federated Multi-Task Learning
By Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, Ameet Talwalkar. 2017
2,150 citations recorded by the source corpus.
Federated learning poses new statistical and systems challenges in training machine learning models over distributed networks of devices. In this work, we show that multi-task learning is naturally suited to handle the statistical challenges of this setting, and propose a novel systems-aware optimization method, MOCHA, that is robust to practical systems issues. Our method and theory for the first time consider issues of high communication cost, stragglers, and fault tolerance for distributed multi-task learning. The resulting method achieves significant speedups compared to alternatives in the federated setting, as we demonstrate through simulations on real-world federated datasets.
Subfields: Artificial Intelligence, Building and Construction, Computer Networks and Communications, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Qiang Yang
- Dacheng Tao
- Martin Jaggi
- Percy Liang
- Peter Richtárik
- Yoshua Bengio
- Bernt Schiele
- Mohit Bansal
In Building and Construction
- Aryan Mokhtari
- Bingsheng He
- Qinbin Li
- Virginia Smith
- Tian Li
- Ameet Talwalkar
- Chaoyang He
- Luping Wang
In Computer Networks and Communications
- Medhi Bennis
- Merouane Debbah
- Martin Jaggi
- Walid Saad
- Aryan Mokhtari
- Khaled Ben Letaief
- Jihong Park
- H. Vincent Poor
In Federated & Distributed Learning
- Virginia Smith
- Jianyu Wang
- Aryan Mokhtari
- Jakub Konecný
- Gauri Joshi
- Martin Jaggi
- Qiang Yang
- Bingsheng He
Citations in the corpus
- Advances and Open Problems in Federated Learning (2019, methodology): “[362] introduced the MOCHA algorithm for multi-task federated learning, directly tackling challenges of communication efficiency, stragglers, and fault tolerance.”
- Federated Optimization in Heterogeneous Networks (2018, methodology): “Federated learning methods (e.g., McMahan et al., 2017; Smith et al., 2017) are designed to handle multiple devices collecting data and a central server coordinating the global learning objective across the network.”
- Ensemble Distillation for Robust Model Fusion in Federated Learning (2020, background): “Federated Learning (FL) has emerged as an important machine learning paradigm in which a federation of clients participate in collaborative training of a centralized model (Shokri & Shmatikov, 2015; McMahan et al., 2016; Smith et al., 2017; Caldas et al., 2018; Bonawitz et al., 2019; Li et al., 2019”
- Flower: A Friendly Federated Learning Research Framework (2020, background): “Finally, there is an increasing emphasis on the performance of federated optimization in heterogeneous data and system settings (Smith et al., 2017; Li et al., 2018; 2019).”
- Personalized Federated Learning with Moreau Envelopes (2020, methodology): “Finally, multi-task learning can be used for personalization: [20] introduced a federated multi-task framework”
- Three Approaches for Personalization with Applications to Federated Learning (2020, background): “Personalization in the context of FL has been studied by several works via multi-task learning (Smith et al., 2017), meta-learning (Jiang et al.”
- Personalized Federated Learning: A Meta-Learning Approach (2020, methodology): “In addition, Smith et al. (2017) used multi-task learning framework and proposed a new method, MOCHA, to address these statistical and systems challenges (including data heterogeneity as well as communication efficiency).”
- Salvaging Federated Learning by Local Adaptation (2020, background): “Training of participant-specific models is studied in (Smith et al., 2017), without privacy or robustness and at the cost of re-placing the entire federated learning framework.”
- Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning (2020, background): “A multi-task learning framework for federated learning has been proposed in [41], for learning several models simultaneously.”
- Robust Aggregation for Federated Learning (2019, background): “Extensions were proposed in [7, 29, 33, 47, 50, 57, 69, 75, 79]; see also the recent surveys [46, 56].”
- Secure Federated Submodel Learning (2019, neutral): “Different from the above work, Smith et al. [35] focused on learning separate but related personalized models for distinct clients by leveraging multitask learning for shared representation.”
- Communication Efficient Decentralized Training with Multiple Local Updates (2019, background): “An orthogonal one is to pay more local computation for less communication, e.g., one-shot aggregation [53, 54, 16, 21, 46], primal-dual algorithms [37, 38, 7] and distributed Newton methods [33, 52, 29, 35, 25].”
- Central Server Free Federated Learning over Single-sided Trust Social Networks (2019, background): “To address both statistical and system challenges, Smith et al. (2017b) and Caldas et al. (2018) propose a multi-task learning framework for federated learning and its related optimization algorithm, which extends early works SDCA (Shalev-Shwartz and Zhang, 2013; Yang, 2013; Yang et al., 2013) and…”
- The Non-IID Data Quagmire of Decentralized Machine Learning (2019, background): “For example, instead of training a global model to fit non-IID data partitions, federated multi-task learning (Smith et al., 2017) proposes training local models for each data partition while leveraging other data partitions to improve the model accuracy.”
- FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization (2019, methodology): “Examples of such applications are learning from wearable devices [Huang et al., 2018], learning sentiment [Smith et al., 2017], and location-based services [Samarakoon et al., 2018].”
- Differentially Private Meta-Learning (2019, background): “That being said, a federated setting involving per-user personalization (Chen et al., 2018; Smith et al., 2017) is a natural meta-learning application.”
- On the Convergence of FedAvg on Non-IID Data (2019, background): “The primal-dual algorithms [6, 26, 27] are more generally applicable and more relevant to FL.”
- Adaptive Gradient-Based Meta-Learning Methods (2019, background): “Federated Learning: A main goal in this setting is to use data on heterogeneous nodes to learn a global model without much communication; leveraging this to get a personalized model is an auxiliary goal [49], with a common application being next-character prediction on mobile devices.”
- Fair Resource Allocation in Federated Learning (2019, background): “reduce communication, as well as to tolerate heterogeneity, methods that allow for local updating and low participation among devices have become de facto solvers for this setting (Li et al., 2018; McMahan et al., 2017; Smith et al., 2017).”