Asynchronous Multi-task Learning

By Inci M. Baytas, Ming Yan, Anil K. Jain, Jiayu Zhou. 2016

47 citations recorded by the source corpus.

Many real-world machine learning applications involveseveral learning tasks which are inter-related. For example, in healthcare domain, we need to learn a predictive model of a certain disease for many hospitals. The models for each hospital may be different because of the inherent differences in the distributions of the patient populations. However, the models are also closely related because of the nature of the learning tasks modeling the same disease. By simultaneously learning all the tasks, multi-task learning (MTL) paradigm performs inductive knowledge transfer among tasks to improve the generalization performance. When datasets for the learning tasks are stored at different locations, it may not always be feasible to transfer the data to provide a data centralized computing environment due to various practical issues such as high data volume and privacy. In this paper, we propose a principled MTL framework for distributed and asynchronous optimization to address the aforementioned challenges. In our framework, gradient update does not wait for collecting the gradient information from all the tasks. Therefore, the proposed method is very efficient when the communication delay is too high for some task nodes. We show that many regularized MTL formulations can benefit from this framework, including the low-rank MTL for sharedsubspace learning. Empirical studies on both synthetic and realworld datasets demonstrate the efficiency and effectiveness of the proposed framework.

Subfields: Artificial Intelligence, Computational Mechanics, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Electrical and Electronic Engineering

Citations in the corpus

Fully Decentralized Joint Learning of Personalized Models and Collaboration Graphs (2019, background): “Recently, distributed and federated learning approaches (Wang et al., 2016b,a; Baytas et al., 2016; Smith et al., 2017) have been proposed to overcome these limitations.”
Federated Multi-Task Learning (2017, background): “The works of [22] and [6] allow for asynchronous updates to help mitigate stragglers, but do not address fault tolerance.”
Multi-Agent Distributed Lifelong Learning for Collective Knowledge Acquisition (2017, background): “Only recently have MTL scenarios been investigated where the tasks are distributed [2, 13, 19, 21, 32, 34].”
Towards Taming the Resource and Data Heterogeneity in Federated Learning (2019, background): “Similarly, [3, 10] allow uniform sampling of the data parties and provide performance guarantee for convex loss functions.”
Asynchronous Online Federated Learning for Edge Devices with Non-IID Data (2019, methodology): “Dynamic learning step sizes are used in asynchronous optimization to achieve better learning performance [35], [36].”
A New Primal–Dual Algorithm for Minimizing the Sum of Three Functions with a Linear Operator (2016, methodology): “as forward–backward splitting [29], backward–forward splitting [2,31], Peaceman–Rachford splitting [16,30], and forward-Douglas–Rachford splitting [3].”
Privacy-Preserving Distributed Multi-Task Learning with Asynchronous Updates (2017, background): “In [4], Baytas et al. proposed an asynchronous MTL framework where the server does not wait for the task nodes to complete their computations before the proximal update of the model is performed.”
SAFA: A Semi-Asynchronous Protocol for Fast Federated Learning With Low Overhead (2019, methodology): “n the basis of FedAvg [3] for achieving fast, communication-efﬁcient federated optimization. SAFA takes advantage of several efﬁciency-boosting featuresfromasynchronousmachinelearningapproaches(e.g., [10][11][12]) while we make use of reﬁned steering control mechanism to mitigate the impact on the g”
Distributed Multi-Task Relationship Learning (2017, methodology): “Different from prior solutions to distributed MTL, which are focused on the former category [5, 29, 30], our proposed DMTRL falls into the latter category.”
Learning Transferable Knowledge Through Embedding Spaces (2019, background): “Only recently have MTL scenarios been investigated where the tasks are distributed [235; 249; 250; 251; 252; 253].”
Asynchronous Online Federated Learning for Edge Devices (2019, methodology): “Dynamic learning step sizes are used in asynchronous optimization to achieve better learning performance [18, 11].”