Multi-task Learning with Labeled and Unlabeled Tasks

By Anastasia Pentina, Christoph H. Lampert. 2016

42 citations recorded by the source corpus.

In multi-task learning, a learner is given a collection of prediction tasks and needs to solve all of them. In contrast to previous work, which required that annotated training data is available for all tasks, we consider a new setting, in which for some tasks, potentially most of them, only unlabeled training data is provided. Consequently, to solve all tasks, information must be transferred between tasks with labels and tasks without labels. Focusing on an instance-based transfer method we analyze two variants of this setting: when the set of labeled tasks is fixed, and when it can be actively selected by the learner. We state and prove a generalization bound that covers both scenarios and derive from it an algorithm for making the choice of labeled tasks (in the active case) and for transferring information between the tasks in a principled way. We also illustrate the effectiveness of the algorithm by experiments on synthetic and real data.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

A Survey on Multi-Task Learning (2017, methodology): “Different from these works, in [132], a derived generalization bound is used to select a subset from multiple unlabeled tasks to acquire labels to improve the generalization performance of all the tasks.”
Taskonomy: Disentangling Task Transfer Learning (2018, methodology): “Multi-task learning has experienced recent progress and the reported advantages are another support for existence of a useful structure among tasks [93, 100, 50, 76, 73, 50, 18, 97, 61, 11, 66].”
Which Tasks Should Be Learned Together in Multi-task Learning? (2019, methodology): “Hard parameter sharing is also used in many other works such as [42, 4, 27, 12, 17, 2, 33, 10, 45, 19, 23, 8, 47, 36].”
Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System (2019, background): “Pentina and Lampert [22] studied a variant of multi-task learning in which annotated data was available on some of the tasks.”
Robust Learning from Untrusted Sources (2019, methodology): “Our ﬁrst set of experiments is on the "Multitask dataset of product reviews" 2 (Pentina & Lampert, 2017), containing customer reviews for 957 Amazon products from the "Amazon product data" (McAuley et al., 2015a;b), together with a binary label indicating whether each review is positive or negative.”
On Localized Discrepancy for Domain Adaptation (2020, background): “Multitask learning problem concentrates on how to share knowledge between different tasks for better overall results Maurer (2006); Pentina and Ben-David (2015); Maurer et al. (2016); Pentina and Lampert (2017); Wu et al. (2020).”