Improving In-Context Few-Shot Learning via Self-Supervised Training

By Mingda Chen, Jingfei Du, Ramakanth Pasunuru, Todor Mihaylov, Srini Iyer, Ves Stoyanov, Zornitsa Kozareva. 2022

82 citations recorded by the source corpus.

Self-supervised pretraining has made few-shot learning possible for many NLP tasks. But the pretraining objectives are not typically adapted specifically for in-context few-shot learning. In this paper, we propose to use self-supervision in an intermediate training stage between pretraining and downstream few-shot usage with the goal to teach the model to perform in-context few shot learning. We propose and evaluate four self-supervised objectives on two benchmarks. We find that the intermediate self-supervision stage produces models that outperform strong baselines. Ablation study shows that several factors affect the downstream performance, such as the amount of training data and the diversity of the self-supervised objectives. Human-annotated cross-task supervision and self-supervision are complementary. Qualitative analysis suggests that the self-supervised-trained models are better at following task requirements.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Prompting & Reasoning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yang Liu
2. Dilek Z. Hakkani-Tür
3. Dimitris Papailiopoulos
4. Jy-yong Sohn
5. Siva Reddy
6. Zhifang Sui
7. Lei Li
8. Nicholas Meade

In Computer Vision and Pattern Recognition

1. Damai Dai
2. Shashank Rajput
3. Xu Sun
4. Dilek Z. Hakkani-Tür
5. Lei Li
6. Siva Reddy
7. Prakhar Gupta
8. Zhiyong Wu

In Large Language Models

1. Damai Dai
2. Xu Sun
3. Dilek Z. Hakkani-Tür
4. Lei Li
5. Siva Reddy
6. Prakhar Gupta
7. Nicholas Meade
8. Kangwook Lee

In Prompting & Reasoning

1. Damai Dai
2. Xu Sun
3. Lei Li
4. Siva Reddy
5. Qingxiu Dong
6. Ce Zheng
7. Zhifang Sui
8. Baobao Chang

In Representation & Self-Supervised Learning

1. Shashank Rajput
2. Xu Sun
3. Dilek Z. Hakkani-Tür
4. Lei Li
5. Siva Reddy
6. Prakhar Gupta
7. Zhiyong Wu
8. Dimitris Papailiopoulos

Citations in the corpus

• Using In-Context Learning to Improve Dialogue Safety (2023, background): “1) Meta In-Context Learning: Meta in-context learning (Min et al., 2022; Chen et al., 2022; Chung et al., 2022; Longpre et al., 2023) has shown to be an effective approach for enabling models to better adapt to novel examples at inference-time.”
• A Survey for In-context Learning (2023, methodology): “To utilize raw corpus for warmup, Chen et al. (2022a) proposed to construct self-supervised training data according to the ICL formats in the down-stream tasks.”
• LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, methodology): “Fine-tuning [66] pretrained LMs is the standard practice for learning downstream tasks, which may involve simple architecture modifi(1)The recent work [60] demonstrates the usefulness of the intermediate fine-tuning method for LMs.”
• A Survey on In-context Learning (2022, methodology): “While the vanilla GPT-3 model itself shows promising ICL abilities, several studies observed that the ability could be significantly boosted via adaption during pretraining (Min et al., 2022b; Chen et al., 2022c).”