Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand

By Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Lavinia Dunagan, Jacob Daniel Morrison, Alexander R. Fabbri, Yejin Choi, Noah A. Smith. 2021

46 citations recorded by the source corpus.

Natural language processing researchers have identified limitations of evaluation methodology for generation tasks, with new questions raised about the validity of automatic metrics and of crowdworker judgments. Meanwhile, efforts to improve generation models tend to depend on simple n-gram overlap metrics (e.g., BLEU, ROUGE). We argue that new advances on models and metrics should each more directly benefit and inform the other. We therefore propose a generalization of leaderboards, bidimensional leaderboards (Billboards), that simultaneously tracks progress in language generation models and metrics for their evaluation. Unlike conventional unidimensional leaderboards that sort submitted systems by predetermined metrics, a Billboard accepts both generators and evaluation metrics as competing entries. A Billboard automatically creates an ensemble metric that selects and linearly combines a few metrics based on a global analysis across generators. Further, metrics are ranked based on their correlation with human judgments. We release four Billboards for machine translation, summarization, and image captioning. We demonstrate that a linear ensemble of a few diverse metrics sometimes substantially outperforms existing metrics in isolation. Our mixed-effects model analysis shows that most automatic metrics, especially the reference-based ones, overrate machine over human generation, demonstrating the importance of updating metrics as generation models become stronger (and perhaps more similar to humans) in the future.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks, Human-Computer Interaction, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

In Human-Computer Interaction

In Signal Processing

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “, 2021) aims to improve data alongside models, and Bidimensional Leaderboards (Kasai et al., 2022) for improving metrics alongside models.”
TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering (2023, background): “Moreover, evaluating text (caption) generation is inherently challenging [23, 26].”
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022, background): “This issue is particularly pertinent in natural language generation (NLG) since the model quality cannot be estimated using accuracy and instead, NLG relies on automatic and human evaluation approaches that constantly improve (Gehrmann et al., 2022; Kasai et al., 2022).”
Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022, methodology): “However, the evaluation process is still dominated by traditional similarity-based metrics (Kasai et al., 2021), exemplified by ROUGE (Lin, 2004) and BLEU (Pa- 1https://github.com/maszhongming/UniEval pineni et al., 2002) that compute n-gram overlap between the model output and the reference text.”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, background): “Other studies analyze a broader set of metrics (Peyrard, 2019; Bhandari et al., 2020; Deutsch and Roth, 2020; Fabbri et al., 2022a; Gabriel et al., 2021; Kasai et al., 2022b), including those specific to factual consistency evaluation (Kryscinski”
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (2023, background): “It has been shown that more than 60% of recent papers on NLG only rely on ROUGE or BLEU to evaluate their systems (Kasai et al., 2021).”
One Embedder, Any Task: Instruction-Finetuned Text Embeddings (2022, methodology): “Similar to summarization evaluation in MTEB, we use the Billboard benchmark (Kasai et al., 2022a) to apply INSTRUCTOR to automatic evaluations for three additional text generation tasks: MSCOCO image captioning (Lin et al.”