GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation

By Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, Woomyeong Park. 2021

290 citations recorded by the source corpus.

Large-scale language models such as GPT-3 are excellent few-shot learners, allowing them to be controlled via natural text prompts. Recent studies report that prompt-based direct classification eliminates the need for fine-tuning but lacks data and inference scalability. This paper proposes a novel data augmentation technique that leverages large-scale language models to generate realistic text samples from a mixture of real samples. We also propose utilizing soft-labels predicted by the language models, effectively distilling knowledge from the large-scale language models and creating textual perturbations simultaneously. We perform data augmentation experiments on diverse classification tasks and show that our method hugely outperforms existing text augmentation methods. Ablation studies and a qualitative analysis provide more insights into our approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Pengfei Liu
2. Graham Neubig
3. Weizhe Yuan
4. Jinlan Fu
5. Jiawei Han
6. Zhengbao Jiang
7. Jung-Woo Ha
8. Yu Meng

In Computer Vision and Pattern Recognition

1. Zhengbao Jiang
2. Pengfei Liu
3. Jinlan Fu
4. Graham Neubig
5. Dongju Park
6. Jiawei Han
7. Yu Meng
8. Laura Schmid

In Natural Language Processing

1. Pengfei Liu
2. Graham Neubig
3. Jinlan Fu
4. Weizhe Yuan
5. Zhengbao Jiang
6. Jiawei Han
7. Jiaxin Huang
8. Yu Zhang

Citations in the corpus

• Large Language Models Can Self-Improve (2022, neutral): “Previous work (Yoo et al., 2021; Meng et al., 2022) discuss few-shot data augmentation by generating diverse training samples using LLMs.”
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, methodology): “[145] propose to generate ameta-prompt based on these answered prompts using prompting methods.”
• Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “There have been studies with similar goals, such as generating similar/dissimilar sentences for training sentence embeddings [58] and using labeled samples as demonstrations to prompt large PLMs [81] for creating novel training data.”
• What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers (2021, background): “The outputs by one large-scale LM can also be input to small-scale LMs (Yoo et al., 2021).”
• Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond (2023, background): “The human-feedback procedure plays an important role in acquiring such abilities LLMs can both act as a good annotator and data generator for data augmentation, such as in[27, 29, 99, 121, 122].”
• A Survey on Data Augmentation for Text Classification (2021, unknown): “22 [114] GPT-3 with prompt-based generation and pseudo-labeling BERT (base)”
• A Holistic Approach to Undesired Content Detection in the Real World (2022, unknown): “Generating synthetic data via large pre-trained language models has shown to be an effective way for data augmentation (Anaby-Tavor et al. 2020; Kumar, Choudhary, and Cho 2020; Yoo et al. 2021) and it is particularly helpful when there is little to no initial data (“cold start”) or when there are no”
• Text Data Augmentation for Deep Learning (2021, methodology): “Drawing inspiration from the success of MixUp, which was presented in further detail in MixUp Augmentation, Yoo et al. developed GPT3Mix [69].”
• AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025, unknown): “The emergence of AI Agent as a transformative paradigm in artificial intelligence is closely tied to the evolution and repurposing of large-scale language models such as GPT-3 [120], Llama [121], T5 [122], Baichuan 2 [123] and GPT3mix [124].”
• Large Language Models Are Reasoning Teachers (2022, background): “Diverse reasoning also draws parallels with Yoo et al. (2021) which utilizes the generative power of LLMs to augment training data with synthesized samples.”
• PERSONACHATGEN: Generating Personalized Dialogues using GPT-3 (2022, background): “Recently, as an alternate way, many studies have created datasets by leveraging pre-trained language models with designed prompt instructions (Yoo et al., 2021; Baheti et al., 2021; Hartvigsen et al., 2022) due to their enormous ability to produce more human-like text (Clark et al., 2021; Dou et…”
• Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning (2022, methodology): “Only one example is shown in the template for clarity; in practice, we concatenate k = 4 samples according to the optimal setting in GPT3Mix (Yoo et al., 2021).”
• Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review (2023, background): “Besides, [130] is designed for the generation of supplementary synthetic data for classification tasks.”
• Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations (2022, background): “For example, few-shot ICL models can be improved to tolerate label noises from synthetic data samples generated in the joint input and label spaces (Yoo et al., 2021).”
• Data-centric Artificial Intelligence: A Survey (2023, unknown): “As models become sufficiently powerful, it becomes feasible to perform many data-centric AI tasks with models, such as data labeling [161], 129:28 D. Zha et al. and data augmentation [250].”