Dynabench: Rethinking Benchmarking in NLP

By Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Pratik Ringshia, Zhiyi Ma and 8 more. 2021

580 citations recorded by the source corpus.

We introduce Dynabench, an open-source platform for dynamic dataset creation and model benchmarking. Dynabench runs in a web browser and supports human-and-model-in-the-loop dataset creation: annotators seek to create examples that a target model will misclassify, but that another person will not. In this paper, we argue that Dynabench addresses a critical need in our community: contemporary models quickly achieve outstanding performance on benchmark tasks but nonetheless fail on simple challenge examples and falter in real-world scenarios. With Dynabench, dataset creation, model development, and model assessment can directly inform each other, leading to more robust and informative benchmarks. We report on four initial NLP tasks, illustrating these concepts and highlighting the promise of the platform, and address potential objections to dynamic benchmarking as a new standard for the field.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “, DynaBench (Kiela et al., 2021) aims to improve data alongside models, and Bidimensional Leaderboards (Kasai et al.”
HateCheck: Functional Tests for Hate Speech Detection Models (2020, background): “This could be addressed by “live” datasets, such as dynamic adversarial benchmarks (Nie et al., 2020; Vidgen et al., 2020b; Kiela et al., 2021).”
Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, methodology): “In addition to incorporating an efficiency measure as part of leader boards for various ML tasks, data [100], models7, training algorithms [101], environmental impact must also be considered and adopted by AI system hardware developers.”
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, background): “Others use humans in the loop to continuously and dynamically build, break, and ﬁx [20] models in order to continuously make them more robust to failure modes [40, 32, 55, 61].”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “Evolving benchmarks, such as Dynabench (Kiela et al., 2021), or other adversarial iterative procedures (Dinan et al., 2019b; Nie et al., 2019; Xu et al., 2020) can provide the required adaptability: our societal standards and expectations change, and we would not tolerate models that do not reﬂect t”
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “Notably, DynaBench [21], a research platform dedicated to dynamic data collection and benchmarking, aligns with our spirit.”
CommonsenseQA 2.0: Exposing the Limits of AI through Gamification (2021, methodology): “Similar to prior work [21, 22, 23], a model is trained on collected examples, which prevents players from using the same types of questions repeatedly, as the re-trained model quickly adapts to such questions.”
Adversarial Training for High-Stakes Reliability (2022, background): “Several recent datasets and benchmarks are constructed with human-in-the-loop adversaries, such AdversarialNLI [39], AdversarialGLUE [40], and DynaBench [41].”
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants (2021, methodology): “We use an adaptation of the Dynabench (Kiela et al., 2021) QA interface that allows annotators to interact with the models in the loop, and further allows them to edit and modify generated questions and answers as required.”
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline (2024, background): “DynaBench [24] identifies these challenges and recommends creating living and continuously evolving benchmarks.”
CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022, background): “Then, we describe potential use cases of CoAuthor in HCI and NLP research, exemplifying the potential of CoAuthor serving as a boundary object.”
Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review (2023, methodology): “This method tries to evaluate the performance of LLMs on various benchmark datasets [108, 109] and demonstrate their efficiency.”