Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias

By Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander J. Ratner, Ranjay Krishna, Jiaming Shen, Chao Zhang. Neural Information Processing Systems, 2023

367 citations recorded by the source corpus.

Large language models (LLMs) have been recently leveraged as training data generators for various natural language processing (NLP) tasks. While previous research has explored different approaches to training models using generated data, they generally rely on simple class-conditional prompts, which may limit the diversity of the generated data and inherit systematic biases of LLM. Thus, we investigate training data generation with diversely attributed prompts (e.g., specifying attributes like length and style), which have the potential to yield diverse and attributed generated data. Our investigation focuses on datasets with high cardinality and diverse domains, wherein we demonstrate that attributed prompts outperform simple class-conditional prompts in terms of the resulting model's performance. Additionally, we present a comprehensive empirical study on data generation encompassing vital aspects like bias, diversity, and efficiency, and highlight three key observations: firstly, synthetic datasets generated by simple prompts exhibit significant biases, such as regional bias; secondly, attribute diversity plays a pivotal role in enhancing model performance; lastly, attributed prompts achieve the performance of simple class-conditional prompts while utilizing only 5\% of the querying cost of ChatGPT associated with the latter. The data and code are available on \url{https://github.com/yueyu1030/AttrPrompt}.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/78c488e2d84bd193a40006b1fceb03e3845b81d4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qun Liu
2. Xin Jiang
3. Lifeng Shang
4. Tyler Derr
5. Nedim Lipka
6. Ryan A. Rossi
7. Ruiyi Zhang
8. Liangyou Li

Citations in the corpus

• Knowledge Graph Prompting for Multi-Document Question Answering (2023, background): “Pre-train, Prompt, and Predict with LLMs With the emergence of LLMs, the paradigm of ‘pre-train, prompt, predict’ has gained magnificent popularity in handling a wide spectrum of tasks (Gururangan et al. 2020; Liu et al. 2023; Yu et al. 2023).”
• Aligning Large Language Models with Human: A Survey (2023, background): “, 2023c), AttrPrompt (Yu et al., 2023b), WizardLM (Xu et al.”