Distributed Multi-Task Relationship Learning

By Sulin Liu, Sinno Jialin Pan, Qirong Ho. 2017

75 citations recorded by the source corpus.

Multi-task learning aims to learn multiple tasks jointly by exploiting their relatedness to improve the generalization performance for each task. Traditionally, to perform multi-task learning, one needs to centralize data from all the tasks to a single machine. However, in many real-world applications, data of different tasks may be geo-distributed over different local machines. Due to heavy communication caused by transmitting the data and the issue of data privacy and security, it is impossible to send data of different task to a master machine to perform multi-task learning. Therefore, in this paper, we propose a distributed multi-task learning framework that simultaneously learns predictive models for each task as well as task relationships between tasks alternatingly in the parameter server paradigm. In our framework, we first offer a general dual form for a family of regularized multi-task relationship learning methods. Subsequently, we propose a communication-efficient primal-dual distributed optimization algorithm to solve the dual problem by carefully designing local subproblems to make the dual problem decomposable. Moreover, we provide a theoretical convergence analysis for the proposed algorithm, which is specific for distributed multi-task relationship learning. We conduct extensive experiments on both synthetic and real-world datasets to evaluate our proposed framework in terms of effectiveness and convergence.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

Citations in the corpus

Federated Multi-Task Learning (2017, methodology): “Compared to [49, 30] which assumes θ t = θ < 1, our assumption is significantly less restrictive and better models the federated setting, where nodes are unreliable and may periodically drop out.”
Data Poisoning Attacks on Multi-Task Relationship Learning (2018, methodology): “Speciﬁcally, we focus on multi-task relationship learning (MTRL) models, a popular subclass of MTL models where task relationships are quantized and are learned directly from training data (Zhang and Yeung 2010; Liu, Pan, and Ho 2017).”
Zero-Shot Task Transfer (2019, background): “Distributed multi-task learning methods [25] address the same objective when tasks are distributed across a network.”
Personalized Federated Learning With Differential Privacy (2020, background): “Besides, some works recently studied the problem of collaborative learning of personalized models, however, they did not address any privacy issue [7], [24], [27].”
Privacy-Preserving Distributed Multi-Task Learning with Asynchronous Updates (2017, background): “Recently, Liu et al. [34] proposed a distributed MTL framework that can simultaneously learn task relationships and task-specific predictive models from geo-distributed task data.”
Lower Bounds and Optimal Algorithms for Personalized Federated Learning (2020, methodology): “In this work, we focus on the mixing FL objective from [19], which is well-known from the area of distributed optimization [26, 16] and distributed transfer learning [30, 50].”
Learning Transferable Knowledge Through Embedding Spaces (2019, background): “Only recently have MTL scenarios been investigated where the tasks are distributed [235; 249; 250; 251; 252; 253].”
Knowledge Aggregation via Epsilon Model Spaces (2018, unknown): “Federated averaging has also been explored in multi-task settings [23; 17].”
Federated Kernelized Multi-Task Learning (2018, methodology): “Finally, previous works have either argued for [10, 17] or empirically demonstrated [13] the benefits of introducing non-linear mappings in their MTL formulations.”
Branched Multi-Task Networks: Deciding what layers to share (2019, background): “Early work on MTL often relied on sparsity constraints [4, 19, 27, 30, 45]”
Federated learning with hierarchical clustering of local updates to improve training on non-IID data (2020, unknown): “By considering each local learning objective as a task, multi-task learning [14] has been employed in the distributed learning setting in multiple works [15]–[17].”
Multi-Agent Distributed Lifelong Learning for Collective Knowledge Acquisition (2017, background): “Only recently have MTL scenarios been investigated where the tasks are distributed [2, 13, 19, 21, 32, 34].”