GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
By Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Anna Shvets, Ashish Upadhyay, Bingsheng Yao and 66 more. 2022
43 citations recorded by the source corpus.
Evaluation in machine learning is usually informed by past choices, for example which datasets or metrics to use. This standardization enables the comparison on equal footing using leaderboards, but the evaluation choices become sub-optimal as better alternatives arise. This problem is especially pertinent in natural language generation which requires ever-improving suites of datasets, metrics, and human evaluation to make definitive claims. To make following best model evaluation practices easier, we introduce GEMv2. The new version of the Generation, Evaluation, and Metrics Benchmark introduces a modular infrastructure for dataset, model, and metric developers to benefit from each others work. GEMv2 supports 40 documented datasets in 51 languages. Models for all datasets can be evaluated online and our interactive data card creation and rendering tools make it easier to add new datasets to the living benchmark.
Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Mohit Bansal
- Yejin Choi
- Noah A. Smith
- Yonatan Belinkov
- Percy Liang
- Graham Neubig
- Jianfeng Gao
In Natural Language Processing
- Mohit Bansal
- Yonatan Belinkov
- Yejin Choi
- Noah A. Smith
- Sebastian Gehrmann
- Graham Neubig
- Samuel R. Bowman
- Percy Liang
In Evaluation & Benchmarks
- Sebastian Gehrmann
- Verena Rieser
- Yonatan Belinkov
- Jekaterina Novikova
- Ehud Reiter
- Miruna Clinciu
- Ondrej Dusek
- Zdeněk Kasner
Citations in the corpus
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “Progress on the first goal is presented in the work of Jernite et al. (2022), which further motivates the needs and requirements of data governance, and outlines the structure needed for a network 4. bigscience.huggingface.co/blog/bigscience-ethical-charter of data custodians, rights-holders, and…”