WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation

By Alisa Liu, Swabha Swayamdipta, Noah A. Smith, Yejin Choi. 2022

265 citations recorded by the source corpus.

A recurring challenge of crowdsourcing NLP datasets at scale is that human writers often rely on repetitive patterns when crafting examples, leading to a lack of linguistic diversity. We introduce a novel approach for dataset creation based on worker and AI collaboration, which brings together the generative strength of language models and the evaluative strength of humans. Starting with an existing dataset, MultiNLI for natural language inference (NLI), our approach uses dataset cartography to automatically identify examples that demonstrate challenging reasoning patterns, and instructs GPT-3 to compose new examples with similar patterns. Machine generated examples are then automatically filtered, and finally revised and labeled by human crowdworkers. The resulting dataset, WANLI, consists of 107,885 NLI examples and presents unique empirical strengths over existing NLI datasets. Remarkably, training a model on WANLI improves performance on eight out-of-domain test sets we consider, including by 11% on HANS and 9% on Adversarial NLI, compared to training on the 4x larger MultiNLI. Moreover, it continues to be more effective than MultiNLI augmented with other NLI datasets. Our results demonstrate the promise of leveraging natural language generation techniques and re-imagining the role of humans in the dataset creation process.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Self-Instruct: Aligning Language Models with Self-Generated Instructions (2022, background): “A variety of works have relied on generative LMs for data generation (Schick and Schütze, 2021; Wang et al., 2021; Liu et al., 2022; Meng et al., 2022) or augmentation (Feng et al.”
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2022, methodology): “Inspired by recent work on utilizing language models for data generation (Schick and Schütze, 2021b; Lee et al., 2021; Liu et al., 2022a), we collect data in a fully automatic manner by prompting a pretrained language model with three examples from the Super-Natural Instructions2 dataset (Mishra et…”
Self-critiquing models for assisting human evaluators (2022, background): “[LSSC22] use assistance to help humans create demonstrations to create challenging NLI datasets.”
Reframing Human-AI Collaboration for Generating Free-Text Explanations (2021, methodology): “…work has used binary judgements from crowdworkers to ﬁt models to human preferences for non-XAI tasks such as summarization (Ziegler et al., 2020; Stiennon et al., 2020), creating commonsense knowledge bases (West et al., 2021), and building natural language inference datasets (Liu et al., 2022).”
Discovering Language Model Behaviors with Model-Written Evaluations (2022, background): “, 2021), training data (Liu et al., 2022), and adversarial training data (Nie et al.”
Teaching language models to support answers with verified quotes (2022, methodology): “(2021)), and has also been used for the creation of a language dataset in recent work (Liu et al., 2022).”
Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations (2022, methodology): “Following Liu et al. (2022), we order the in-context samples in increasing order of similarity to the test sample such that the most similar sample is last in the context.”
The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks (2022, methodology): “Experimental Results: We use RoBERTa trained on SNLI (RoBERTa-base-SNLI) (Liu et al., 2019), ELMo-based Decomposable Attention (ELMoDA) (Parikh et al., 2016), ALBERT (Lan et al., 2019), distilled version of the RoBERTa-base model (Sanh et al., 2019), and RoBERTa-large finetuned on WANLI (Liu et al.,”
Is GPT-3 a Good Data Annotator? (2022, background): “In a recent work, Liu et al. (2022) demonstrate a worker-and-AI collaborative approach for dataset creation with a few seed examples, while we also analyze approaches that support zero-shot training data generation, which do not require any seed examples.”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, background): “[5] Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi.”
Data Augmentation Approaches in Natural Language Processing: A Survey (2021, background): “Is DA still helpful in the era of pre-trained language models? Or, how to further use the information in pre-trained models to generate more diverse and high-quality data with less cost? There are some initial explorations in these directions [143, 144], while we still look forward to more works in ”
Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias (2023, methodology): “A recent work WANLI [26] also considers human-AI collaboration for creating more challenging training data, but requires an initial dataset and a strong task model.”
Can Foundation Models Wrangle Your Data? (2022, background): “Put simply, we are transitioning to a world of “worker and AI collaboration” [49] in which all communication is done through natural language.”
CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society (2023, background): “LLMs have been explored for data generation in [101, 63, 68, 114], and Self-Instruct [119] proposes a semi-automated process for instruction instance generation.”
CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society (2023, background): “LLMs have been explored for data generation in [59, 38, 40, 66], and Self-Instruct [70] proposes a semi-automated process for instruction instance generation.”
Impossible Distillation for Paraphrasing and Summarization: How to Make High-quality Lemonade out of Small, Low-quality Model (2023, methodology): “For semantic equivalence, we employ an off-the-shelf NLI model (Liu et al., 2022a), and determine a pair ( x, y ) to be semantically equivalent if entailment holds in both directions.”
ZeroGen: Efficient Zero-shot Learning via Dataset Generation (2022, methodology): “Supervised data generation methods are also studied for building auxiliary tasks (Vu et al., 2021) and dataset creation based on human and machine collaboration (Liu et al., 2022).”
Natural Language Deduction through Search over Statement Compositions (2022, background): “…example contains a contradiction, the ‘neutral’ and ‘contradiction’ labels are merged in EBE NTAIL , as both reflect a failure to entail the goal. model (He et al., 2021) fine-tuned on the WANLI dataset (Liu et al., 2022) to predict the probability that each derived statement entails the goal.”
Uncertainty Estimation for Language Reward Models (2022, background): “While prior work has attempted to leverage language models to automate part of the labeling process [48, 25], we are not aware of any work ﬁne-tuning language models that has tried actively selecting points for human feedback.”