Interpretable Multi-dataset Evaluation for Named Entity Recognition

By Jinlan Fu, Pengfei Liu, Graham Neubig. 2020

49 citations recorded by the source corpus.

With the proliferation of models for natural language processing tasks, it is even harder to understand the differences between models and their relative merits. Simply looking at differences between holistic metrics such as accuracy, BLEU, or F1 does not tell us why or how particular methods perform differently and how diverse datasets influence the model design choices. In this paper, we present a general methodology for interpretable evaluation for the named entity recognition (NER) task. The proposed evaluation method enables us to interpret the differences in models and datasets, as well as the interplay between them, identifying the strengths and weaknesses of current systems. By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive progress in this area: this https URL.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “The closest prior work is by Fu et al. (2020) for namedentity recognition which allows similar filtering and sorting, but presents the results in a table.”
ExplainaBoard: An Explainable Leaderboard for NLP (2021, methodology): “…entity (such as “entity frequency,” telling us how well the model does on entities that appear in the training data a certain number of times) or sentences (such as “sentence length,” telling us how well the model does on entities that appear in longer or shorter sentences) (Fu et al., 2020a).”
Polyglot Prompt: Multilingual Multitask PrompTraining (2022, methodology): “Q2: Why does PolyPrompt work? What characteristics of the dataset and language affect the improvement? To answer this question, we design a multilingual interpretable evaluation methodology (Fu et al., 2020; Liu et al., 2021a) to analyze the strengths and weaknesses of the unified framework for diff”
A Primer on Pretrained Multilingual Language Models (2021, unknown): “Evaluation frameworks such as Explainaboard (Liu et al., 2021; Fu et al., 2020) need to also be developed for a range of tasks and languages to identify the nature of errors made by multilingual models.”
DataLab: A Platform for Data Analysis and Intervention (2022, background): “Existing works have shown its advantages in better system designs (Zhong et al., 2019; Fu et al., 2020b; Te-jaswin et al., 2021) .”
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation (2021, methodology): “To provide more granular evaluation capabilities, we extend Fu et al. (2020); Liu et al. (2021)’s EXPLAINABOARD to the task categories and languages in XTREME-R.”
SpanNER: Named Entity Re-/Recognition as Span Prediction (2021, methodology): “To address this problem, we follow the interpretable evaluation idea (Fu et al., 2020a,c) that proposes to breakdown the holistic performance into different buckets from different perspectives and use a performance heatmap to illustrate relative advantages between two systems, i.e., system-pair…”