What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?

By Nikita Nangia, Saku Sugawara, Harsh Trivedi, Alex Warstadt, Clara Vania, Sam Bowman, Samuel R. Bowman. 2021

38 citations recorded by the source corpus.

Crowdsourcing is widely used to create data for common natural language understanding tasks. Despite the importance of these datasets for measuring and refining model understanding of language, there has been little focus on the crowdsourcing methods used for collecting the datasets. In this paper, we compare the efficacy of interventions that have been proposed in prior work as ways of improving data quality. We use multiple-choice question answering as a testbed and run a randomized trial by assigning crowdworkers to write questions under one of four different data collection protocols. We find that asking workers to write explanations for their examples is an ineffective stand-alone strategy for boosting NLU example difficulty. However, we find that training crowdworkers, and then using an iterative process of collecting data, sending feedback, and qualifying workers based on expert judgments is an effective means of collecting challenging data. But using crowdsourced, instead of expert judgments, to qualify workers and send feedback does not prove to be effective. We observe that the data from the iterative protocol with expert assessments is more challenging by several measures. Notably, the human–model gap on the unanimous agreement portion of this data is, on average, twice as large as the gap for the baseline protocol data.

Subfields: Artificial Intelligence, Computer Science Applications, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, background): “Such prompts are generally derived either from crowdsourcing (Nadeem et al., 2021; Nangia et al., 2021) or from slotting a set of terms into templates (Kurita et al.”
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation (2022, methodology): “This follows the recommended crowdsourcing protocol from Nangia et al. (2021).”
QuALITY: Question Answering with Long Input Texts, Yes! (2021, methodology): “Nangia et al. (2021) compare crowdsourcing methods for collecting high-quality QA data and ﬁnd that a long training process with iterative feedback and qualiﬁcations is an effective crowdsourcing strategy.”
Analyzing Dynamic Adversarial Training Data in the Limit (2021, background): “This includes feedback from experts (Parrish et al., 2021; Nangia et al., 2021), gamifying the data collection process (Yang et al.”
CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge (2021, background): “First, we ask crowdworkers to write down the reason why the generated claim is true or false; although past work observes that this does not improve example quality in isolation [Nangia et al., 2021], we found it helpful for our task, and it additionally helped us spot workers who misunderstood the ”
When Combating Hype, Proceed with Caution (2021, background): “8 In practice, research into more effective crowdsourcing and benchmark design (Kiela et al., 2021; Bowman and Dahl, 2021; Nangia et al., 2021; Gehrmann et al., 2021) and research into better statistical reporting and publication norms (Dodge et al.”
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants (2023, methodology): “In a process similar to what Nangia et al. (2021) advo-cates for, we implement an iterative procedure with the Language Service Provider (LSP) for this involved data collection task.”
ANLIzing the Adversarial Natural Language Inference Dataset (2020, background): “, 2021) and analyzed inherent disagreements between human annotators (Pavlick and Kwiatkowski, 2019; Nie et al., 2020b; Nangia et al., 2021), all in the service of understanding”
SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets (2021, background): “This is because collecting high-quality labeled text datasets is slow and expensive, requiring significant labor from skilled human annotators [35].”
CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation (2022, methodology): “Following best practices for crowdsourcing protocols described in the literature (Nangia et al., 2021), we provide personalized feedback to each crowdworker based on their previous round of submissions, describing where”
"I'm sorry to hear that": finding bias in language models with a holistic descriptor dataset (2022, background): “A common approach to measuring bias in both generative and classiﬁcation models relies on prompts generated by seeding crowdworkers with terms and having them write prompts from them (Nadeem et al., 2021; Nangia et al., 2021).”