DynaSent: A Dynamic Benchmark for Sentiment Analysis

By Christopher Potts, Zhengxuan Wu, Atticus Geiger, Douwe Kiela. 2020

87 citations recorded by the source corpus.

We introduce DynaSent (‘Dynamic Sentiment’), a new English-language benchmark task for ternary (positive/negative/neutral) sentiment analysis. DynaSent combines naturally occurring sentences with sentences created using the open-source Dynabench Platform, which facilities human-and-model-in-the-loop dataset creation. DynaSent has a total of 121,634 sentences, each validated by five crowdworkers, and its development and test splits are designed to produce chance performance for even the best models we have been able to develop; when future models solve this task, we will use them to create DynaSent version 2, continuing the dynamic evolution of this benchmark. Here, we report on the dataset creation effort, focusing on the steps we took to increase quality and reduce artifacts. We also present evidence that DynaSent’s Neutral category is more coherent than the comparable category in other benchmarks, and we motivate training models from scratch for each round over successive fine-tuning.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “This can, for example, be seen in Dynabench,1 (Potts et al., 2020) which has a static evaluation, but interactively adds more test 1https://dynabench.org/ data through a human-in-the-loop approach.”
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants (2021, background): “Separately, Ettinger et al. (2017) provide breakers with the ability to minimally edit original data to identify the boundaries of system capabilities, while Potts et al. (2021) analyse the use of prompts to assist crowdworkers in beating a model in the loop for sentiment analysis.”
Dynabench: Rethinking Benchmarking in NLP (2021, background): “Potts et al. (2020) provide an initial report and the ﬁrst two rounds of this dataset.”
Mind the Gap: Assessing Temporal Generalization in Neural Language Models (2021, background): “An open research question is thus how we can create and maintain benchmarks that are not static (Nie et al., 2020; Potts et al., 2020) and further promote research on continual and life-long learning.”
Analyzing Dynamic Adversarial Training Data in the Limit (2021, methodology): “Traditional crowdsourcing methods often yield datasets that lack diversity, contain spurious correlations, and are easy for existing models (Gururangan et al., 2018; Poliak et al., 2018; Geva et al., 2019; Ko et al., 2020; Potts et al., 2021).”
Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation (2021, methodology): “Other datasets for question answering (Rajpurkar et al., 2018; Dua et al., 2019; Wallace et al., 2019), sentiment analysis (Potts et al., 2021), hate speech detection (Vidgen et al., 2021), and natural language inference (Nie et al., 2020) have been collected in a similar manner.”
Can Large Language Models Transform Computational Social Science? (2023, methodology): “Even in the commonly used Majority Vote annotation scheme (Snow et al., 2008; Potts et al., 2021), an LLM can be used as one of multiple annotators to annotate the same amount of data with between 50% and 33% less human labor for a binary task.”
Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022, unknown): “I think will be extremely helpful.”
Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking (2021, methodology): “Evaluation-as-a-Service NLI SNLI [5], MNLI [74] matched and mis-matched, ANLI rounds 1-3 [49] Respective dev sets; HANS [43], NLI stress tests [47], Winogender [62] recast as NLI [53] QA SQuAD dev [57], AdversarialQA [1] (Dynabench QA round 1) AdversarialQA dev, the 12 dev sets from MRQA shared task”
SituatedQA: Incorporating Extra-Linguistic Contexts into QA (2021, background): “Adversarial, phased data collection has been proposed (Paperno et al., 2016; Zellers et al., 2018; Potts et al., 2020) to drive model development, constantly feeding models examples that current models are unable to address.”
On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study (2021, methodology): “Similar strategies were employed in (Ettinger et al., 2017), where breakers minimally edited original data to elicit incorrect predictions from the models built by builders , as well as in recently introduced adversarial benchmarks for sentiment analysis (Potts et al., 2020).”
Pitfalls of Static Language Modelling (2021, background): “…and maintain benchmarks that are not static (e.g., see recent attempt that utilize human feed-back for adversarial creation of benchmarks (Nie et al., 2019; Potts et al., 2020)) and ﬁxed at a particular point in time, but rather are updated online and reﬂect the non-stationarity of the real world.”
RealTime QA: What's the Answer Right Now? (2022, background): “Several other prior works update challenge test sets (Kiela et al., 2021; Potts et al., 2021; Ma et al., 2021), evaluation tasks (Thrush et al., 2022), or text generation metrics (Gehrmann et al., 2021, 2022; Mishra and Arunkumar, 2021; Kasai et al., 2022).”