Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative

By Lucio M. Dery, Paul Michel, Ameet Talwalkar, Graham Neubig. 2021

40 citations recorded by the source corpus.

In most settings of practical concern, machine learning practitioners know in advance what end-task they wish to boost with auxiliary tasks. However, widely used methods for leveraging auxiliary data like pre-training and its continued-pretraining variant are end-task agnostic: they rarely, if ever, exploit knowledge of the target task. We study replacing end-task agnostic continued training of pre-trained language models with end-task aware training of said models. We argue that for sufficiently important end-tasks, the benefits of leveraging auxiliary data in a task-aware fashion can justify forgoing the traditional approach of obtaining generic, end-task agnostic representations as with (continued) pre-training. On three different low-resource NLP tasks from two domains, we demonstrate that multi-tasking the end-task and auxiliary objectives results in significantly better downstream task performance than the widely-used task-agnostic continued pre-training paradigm of Gururangan et al. (2020). We next introduce an online meta-learning algorithm that learns a set of multi-task weights to better balance among our multiple auxiliary objectives, achieving further improvements on end-task performance and data efficiency.

Subfields: Artificial Intelligence, Large Language Models, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bin Liu
2. Tatsuya Konishi
3. Gyuhak Kim
4. Zixuan Ke
5. Yijia Shao
6. Haowei Lin
7. Bing Liu
8. Changnan Xiao

In Large Language Models

1. Bin Liu
2. Tatsuya Konishi
3. Gyuhak Kim
4. Haowei Lin
5. Yijia Shao
6. Zixuan Ke

In Optimization & Training

1. Tatsuya Konishi
2. Gyuhak Kim
3. Haowei Lin
4. Bin Liu
5. Yijia Shao
6. Zixuan Ke

Citations in the corpus

• Continual Pre-training of Language Models (2023, methodology): “…avoid CF while Table 2: End-task macro-F1 (MF1), accuracy and forgetting rate results for all domains after the continual DAP-training of all domains , except for CHEMPORT in the PubMed domain, for which we use micro-F1 following (Gururangan et al., 2020; Dery et al., 2021; Beltagy et al., 2019).”