Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking

By Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Yu Wu, Robin Jia, Christopher Potts, Adina Williams, Douwe Kiela. 2021

71 citations recorded by the source corpus.

We introduce Dynaboard, an evaluation-as-a-service framework for hosting benchmarks and conducting holistic model comparison, integrated with the Dynabench platform. Our platform evaluates NLP models directly instead of relying on self-reported metrics or predictions on a single dataset. Under this paradigm, models are submitted to be evaluated in the cloud, circumventing the issues of reproducibility, accessibility, and backwards compatibility that often hinder benchmarking in NLP. This allows users to interact with uploaded models in real time to assess their quality, and permits the collection of additional metrics such as memory use, throughput, and robustness, which -- despite their importance to practitioners -- have traditionally been absent from leaderboards. On each task, models are ranked according to the Dynascore, a novel utility-based aggregation of these statistics, which users can customize to better reflect their preferences, placing more/less weight on a particular axis of evaluation or dataset. As state-of-the-art NLP models push the limits of traditional benchmarks, Dynaboard offers a standardized solution for a more diverse and comprehensive evaluation of model quality.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “In practice, however, estimating dataset difficulty is often limited to an informal comparison of state-of-the-art model performance to that of humans; the bigger the performance gap, the harder the dataset is said to be (Ethayarajh & Jurafsky, 2020; Ma et al., 2021).”
Analyzing Dynamic Adversarial Training Data in the Limit (2021, methodology): “291 3.3 Dataset Details 292 Our codebase is built on top of the Dynabench 293 platform (Kiela et al., 2021), we deploy tasks us- 294 ing the Mephisto library,3 and we serve models 295 using Dynalab (Ma et al., 2021).”
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards? (2021, methodology): “How to aggregate multi-task benchmarks (Wang et al., 2019b,a; Fisch et al., 2019) and multi-metric benchmarks (Ma et al., 2021) is an open question which—although we do not address—is one use for IRT .”
Perturbation Augmentation for Fairer NLP (2022, methodology): “Previous approaches relied on word lists (Zhao et al., 2019) or designing handcrafted grammars to generate perturbations (Zmigrod et al., 2019; Ma et al., 2021; Ren-duchintala and Williams, 2022; Papakipos and Bitton, 2022).”
Toxicity in Multilingual Machine Translation at Scale (2022, methodology): “…prompts from the beginnings of existing text sentences (Dhamala et al., 2021), and swapping demographic terms in existing text, either heuristically (Ma et al., 2021; Wang et al., 2021; Zhao et al., 2019; Papakipos and Bit-ton, 2022) or using trained neural language models (Qian et al., 2022).”
Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold (2022, background): “Within the NLP community, much of such recent discussion has focused on the nature of leaderboards and the practice of benchmarking (Ethayarajh and Jurafsky, 2020; Ma et al., 2021).”
"I'm sorry to hear that": finding bias in language models with a holistic descriptor dataset (2022, background): “…et al., 2020; Gehman et al., 2020; Huang et al., 2020; Vig et al., 2020; Kirk et al., 2021a; Perez et al., 2022), and some even select existing sentences from text sources and swap demographic terms heuristically (Zhao et al., 2019; Ma et al., 2021; Wang et al., 2021; Papakipos and Bitton, 2022).”
When Combating Hype, Proceed with Caution (2021, background): “Leaderboards that integrate conventional benchmarking with analysis can be especially helpful by making this largely automatic (Wang et al., 2018; Dua et al., 2019; Gehrmann et al., 2021; Ma et al., 2021).”
RealTime QA: What's the Answer Right Now? (2022, background): “Several other prior works update challenge test sets (Kiela et al., 2021; Potts et al., 2021; Ma et al., 2021), evaluation tasks (Thrush et al., 2022), or text generation metrics (Gehrmann et al., 2021, 2022; Mishra and Arunkumar, 2021; Kasai et al., 2022).”
Rethinking Benchmark and Contamination for Language Models with Rephrased Samples (2023, background): “Another approach is to employ dynamic benchmarks (Kiela et al., 2021; Ma et al., 2021), using human-in-the-loop evaluations to reduce the risk of benchmark contamination.”