ExplainaBoard: An Explainable Leaderboard for NLP

By Pengfei Liu, Jinlan Fu, Yanghua Xiao, Weizhe Yuan, Shuaichen Chang, Junqi Dai, Yixin Liu, Zihuiwen Ye, Zi-Yi Dou, Graham Neubig. 2021

61 citations recorded by the source corpus.

With the rapid development of NLP research, leaderboards have emerged as one tool to track the performance of various systems on various NLP tasks. They are effective in this goal to some extent, but generally present a rather simplistic one-dimensional view of the submitted systems, communicated only through holistic accuracy numbers. In this paper, we present a new conceptualization and implementation of NLP evaluation: the ExplainaBoard, which in addition to inheriting the functionality of the standard leaderboard, also allows researchers to (i) diagnose strengths and weaknesses of a single system (e.g. what is the best-performing system bad at?) (ii) interpret relationships between multiple systems. (e.g. where does system A outperform system B? What if we combine systems A, B and C?) and (iii) examine prediction results closely (e.g. what are common errors made by multiple systems or in what contexts do particular errors occur?). So far, ExplainaBoard covers more than 400 systems, 50 datasets, 40 languages, and 12 tasks. We not only released an online platform at the website but also make our evaluation tool an API with MIT Licence at Github and PyPi that allows users to conveniently assess their models offline. We additionally release all output files from systems that we have run or collected to motivate “output-driven” research in the future.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “For example, the the Language Interpretability Tool (Tenney et al., 2020) can automatically evaluate on subpopulations, and Explainaboard (Liu et al., 2021), although with limited support for NLG, helps identifying challenging inputs for a model.”
MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem Solvers (2021, methodology): “…a uniﬁed framework for a certain task, such as OpenNMT (Klein et al., 2017) and TexBox (Li et al., 2021) for text generation tasks, ExplainaBoard (Liu et al., 2021) for evaluating interpretable models, Photon (Zeng et al., 2020) for text-to-SQL tasks, and Huggingface’s Transformers (Wolf et…”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, methodology): “…the probabilities of harmful text detectors directly, composing them with LM output probabilities to up-weight or down-weight the probabilities of each token based on generated left context (Yang and Klein, 2021; Liu et al., 2021a; Dathathri et al., 2019; Krause et al., 2020; Schick et al., 2021).”
BARTScore: Evaluating Generated Text as Text Generation (2021, background): “ai/leaderboard/task-meval/ on the EXPLAINABOARD platform [32], which allows us to interactively understand the strengths, weaknesses, and complementarity of each metric.”
DataLab: A Platform for Data Analysis and Intervention (2022, methodology): “ExplainaBoard (Liu et al., 2021a) presents an analysis platform while it focuses on system diagnostics.”
Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking (2021, background): “• Explainaboard [39] and Robustness Gym [23] do allow for a more nuanced comparison of models, but largely through the lens of which slices of data a model performs well on and what types of errors it makes.”
FacTool: Factuality Detection in Generative AI - A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios (2023, background): “A more granular assessment can be particularly valuable because it (1) not only allows users to pinpoint where inaccuracies occur (Liu et al., 2021) but also (2) serves as a reward”
BRIO: Bringing Order to Abstractive Summarization (2022, methodology): “We calculate 8The calculation is performed using ExplainaBoard (Liu et al., 2021a). https://github.com/neulab/ExplainaBoard.”
Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022, methodology): “A unified evaluator can achieve better extensibility and trans- ferability with continual learning (Parisi et al., 2019) or prompting (Liu et al., 2021b; Chen et al., 2022), as it can accommodate more evaluation dimensions by modifying the input question.”
SpanNER: Named Entity Re-/Recognition as Span Prediction (2021, methodology): “We argue that Leaderboard with an effective system combination (dubbed C OMBINA B OARD ) feature would allow researchers to quickly ﬁnd the complementarities among different systems.”
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization (2021, methodology): “Results of our proposed models have been deployed into EXPLAINABOARD (Liu et al., 2021a) platform, which allows researchers to understand our systems in a more fine-grained way.”
A Primer on Pretrained Multilingual Language Models (2021, unknown): “Evaluation frameworks such as Explainaboard (Liu et al., 2021; Fu et al., 2020) need to also be developed for a range of tasks and languages to identify the nature of errors made by multilingual models.”
Findings of the 2021 Conference on Machine Translation (WMT21) (2021, methodology): “, 2016), and also on ExplainaBoard3 (Liu et al., 2021b).”
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation (2021, methodology): “To provide more granular evaluation capabilities, we extend Fu et al. (2020); Liu et al. (2021)’s EXPLAINABOARD to the task categories and languages in XTREME-R.”
Polyglot Prompt: Multilingual Multitask PrompTraining (2022, methodology): “Prompting Methods Prompting is a technique that aims to make better use of pre-trained knowledge by reformulating tasks at hand (Liu et al., 2021b).”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “Recently, leaderboards like Explainboard [289] which not only evaluate the progress using a single metric like existing benchmarks but also dig deeper by analyzing”