Entity-Switched Datasets: An Approach to Auditing the In-Domain Robustness of Named Entity Recognition Models

By Oshin Agarwal, Yinfei Yang, Byron C. Wallace, Ani Nenkova. 2020

24 citations recorded by the source corpus.

Named entity recognition systems perform well on standard datasets comprising English news. But given the paucity of data, it is difficult to draw conclusions about the robustness of systems with respect to recognizing a diverse set of entities. We propose a method for auditing the in-domain robustness of systems, focusing specifically on differences in performance due to the national origin of entities. We create entity-switched datasets, in which named entities in the original texts are replaced by plausible named entities of the same type but of different national origin. We find that state-of-the-art systems' performance vary widely even in-domain: In the same context, entities from certain origins are more reliably recognized than entities from elsewhere. Systems perform best on American and Indian entities, and worst on Vietnamese and Indonesian entities. This auditing approach can facilitate the development of more robust named entity recognition systems, and will allow research in this area to consider fairness criteria that have received heightened attention in other predictive technology work.

Subfields: Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Re-imagining Algorithmic Fairness in India and Beyond (2021, background): “[12] shows that NLP models disproportionately fail to even detect names of people from non-Western backgrounds.”
Neural Data Augmentation via Example Extrapolation (2021, background): “Ensuring that models make decisions via language understanding rather than memorizing facts or entities has been argued for named entity recognition (Agarwal et al., 2020) and coreference resolution (Agarwal et al.”
Entity-Based Knowledge Conflicts in Question Answering (2021, methodology): “Entity based swap-ping has been used to evaluate robustness in tasks such as coreference resolution (Lu and Ng, 2020) and named entity resolution (Agarwal et al., 2020) as well as to train more robust models (Subrama-nian and Roth, 2019).”
Example-Based Named Entity Recognition (2020, background): “Other studies by Agarwal et al., 2020b expose that while context representations resulting from trained LSTM-CRF and BERT architectures contribute to system performance, the main factor driving high performance is learning name tokens explicitly, which is a weakness when it comes to open NER, when…”
Toward Gender-Inclusive Coreference Resolution: An Analysis of Gender and Bias Throughout the Machine Learning Lifecycle* (2021, background): “One cue that we explored in detail is that around names, and it is worth pointing out recent work by Agarwal et al. (2020) in the context of named entity recognition.”