Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options

By Clara Vania, Ruijie Chen, Samuel R. Bowman. 2020

12 citations recorded by the source corpus.

Large-scale natural language inference (NLI) datasets such as SNLI or MNLI have been created by asking crowdworkers to read a premise and write three new hypotheses, one for each possible semantic relationships (entailment, contradiction, and neutral). While this protocol has been used to create useful benchmark data, it remains unclear whether the writing-based annotation protocol is optimal for any purpose, since it has not been evaluated directly. Furthermore, there is ample evidence that crowdworker writing can introduce artifacts in the data. We investigate two alternative protocols which automatically create candidate (premise, hypothesis) pairs for annotators to label. Using these protocols and a writing-based baseline, we collect several new English NLI datasets of over 3k examples each, each using a fixed amount of annotator time, but a varying number of examples to fit that time budget. Our experiments on NLI and transfer learning show negative results: None of the alternative protocols outperforms the baseline in evaluations of generalization within NLI or on transfer to outside target tasks. We conclude that crowdworker writing still the best known option for entailment data, highlighting the need for further data collection work to focus on improving writing-based annotation processes.

Subfields: Artificial Intelligence, Computer Science Applications, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation (2022, background): “Recent work attempted to remedy this through different data collection protocols but found negative results (Vania et al., 2020; Bowman et al., 2020), showing this is a hard problem requiring greater innovation.”
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021, neutral): “For a relatively simple task like NLI, labeling an existing example likely requires a bare minimum of 45 seconds (Vania et al., 2020), and creating a new example requires at least one minute (Bowman et al., 2020).”
Analyzing Dynamic Adversarial Training Data in the Limit (2021, background): “…(Parrish et al., 2021; Nan- 539 gia et al., 2021), gamifying the data collection 540 process (Yang et al., 2018), encouraging counter- 541 factual examples (Kaushik et al., 2020; Gardner 542 et al., 2020), or providing prompts that workers 543 can edit (Bowman et al., 2020; Vania et al., 2020).”
SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets (2021, background): “Thus human-authored datasets typically consist of relatively short targets [52], and researchers often use automated retrieval-based methods, such as sourcing task examples from the internet, for dataset construction.”
Fool Me Twice: Entailment from Wikipedia Gamification (2021, methodology): “Both Bowman et al. (2020) and Vania et al. (2020) propose alternative methods for collecting entailment pairs from crowdworkers and measure success via improvements in other general tasks via transfer learning.”
ANLIzing the Adversarial Natural Language Inference Dataset (2020, background): “lection methods (Bowman et al., 2020; Vania et al., 2020; Parrish et al., 2021) and analyzed inherent disagreements between human annotators (Pavlick and Kwiatkowski, 2019; Nie et al.”