Unsupervised Cross-Task Generalization via Retrieval Augmentation

By Bill Yuchen Lin, Kangmin Tan, Chris Miller, Beiwen Tian, Xiang Ren. 2022

53 citations recorded by the source corpus.

Humans can perform unseen tasks by recalling relevant skills acquired previously and then generalizing them to the target tasks, even if there is no supervision at all. In this paper, we aim to improve this kind of cross-task generalization ability of massive multi-task language models, such as T0 and FLAN, in an unsupervised setting. We propose a retrieval-augmentation method named ReCross that takes a few unlabelled examples as queries to retrieve a small subset of upstream data and uses them to update the multi-task model for better generalization. ReCross is a straightforward yet effective retrieval method that combines both efficient dense retrieval and effective pair-wise reranking. Our results and analysis show that it significantly outperforms both non-retrieval methods and other baseline methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yejin Choi
2. Hannaneh Hajishirzi
3. Mohit Bansal
4. Luke S. Zettlemoyer
5. Daniel Khashabi
6. Noah A. Smith
7. Swaroop Mishra
8. Jianfeng Gao

In Computer Vision and Pattern Recognition

1. Hannaneh Hajishirzi
2. Yejin Choi
3. Daniel Khashabi
4. Swaroop Mishra
5. Noah A. Smith
6. Bill Yuchen Lin
7. Ximing Lu
8. Chandra Bhagavatula

In Retrieval-Augmented Generation

1. Daniel Khashabi
2. Noah A. Smith
3. Yizhong Wang
4. Hannaneh Hajishirzi
5. Todor Mihaylov
6. Bill Yuchen Lin

Citations in the corpus

• Self-Instruct: Aligning Language Models with Self-Generated Instructions (2022, background): “A series of works have found evidence that vanilla language models can be effective at following general language instructions if tuned with annotated “instructional” data – datasets containing language instructional commands and their desired outcome based on human judgement (Weller et al., 2020; M”
• Improving In-Context Few-Shot Learning via Self-Supervised Training (2022, methodology): “…learning using human-annotated datasets (Ye et al., 2021; Wei et al., 2022; Sanh et al., 2022; Min et al., 2021; Xu et al., 2022) via instructions (Weller et al., 2020; Efrat and Levy, 2020; Mishra et al., 2021; Ouyang et al., 2022) and retrieved examples (Hu et al., 2022; Lin et al., 2022).”
• InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning (2022, methodology): “mance to T0 (Lin et al., 2022) despite being 8 times smaller, whereas the 3B parameter model DIAL-T0 is large and impractical to use on popular affordable GPUs.”
• ToolQA: A Dataset for LLM Question Answering with External Tools (2023, background): “Specifically, one line of research focus on retrieval-augmented language models [58, 3, 17, 27, 30, 80, 34, 72], where they use sparse [54] or dense retrieval [22, 16] to extract relevant knowledge from the corpus.”
• The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (2023, methodology): “In a recent work, Han (2023) demonstrates that in-context learning using approximately 10 dynamic , retrieved examples can achieve impressive performance in aligning base LLMs, sharing a similar motivation with ReCross (Lin et al., 2022).”