SEAL : Interactive Tool for Systematic Error Analysis and Labeling
By Nazneen Rajani, Weixin Liang, Lingjiao Chen, Margaret Mitchell, James Y. Zou. 2022
18 citations recorded by the source corpus.
With the advent of Transformers, large language models (LLMs) have saturated well-known NLP benchmarks and leaderboards with high aggregate performance. However, many times these models systematically fail on tail data or rare groups not obvious in aggregate evaluation. Identifying such problematic data groups is even more challenging when there are no explicit labels (e.g., ethnicity, gender, etc.) and further compounded for NLP datasets due to the lack of visual features to characterize failure modes (e.g., Asian males, animals indoors, waterbirds on land, etc.). This paper introduces an interactive Systematic Error Analysis and Labeling (\seal) tool that uses a two-step approach to first identify high error slices of data and then, in the second step, introduce methods to give human-understandable semantics to those underperforming slices. We explore a variety of methods for coming up with coherent semantics for the error groups using language models for semantic labeling and a text-to-image model for generating visual features. SEAL toolkit and demo screencast is available at https://huggingface.co/spaces/nazneen/seal.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Human-Computer Interaction
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Douwe Kiela
- Mohit Bansal
- Nazneen Rajani
- Tristan Thrush
- Yacine Jernite
- Yejin Choi
- Margaret Mitchell
- Angelina McMillan-Major
In Computer Vision and Pattern Recognition
- Douwe Kiela
- Nazneen Rajani
- Tristan Thrush
- Angelina McMillan-Major
- Yacine Jernite
- Alexandra Sasha Luccioni
- Margaret Mitchell
- Mohit Bansal
In Evaluation & Benchmarks
- Tristan Thrush
- Douwe Kiela
- Angelina McMillan-Major
- Yacine Jernite
- Alexandra Sasha Luccioni
- Margaret Mitchell
- Nazneen Rajani
- Mohit Bansal
In Human-Computer Interaction
Citations in the corpus
- Measuring Data (2022, background): “…for more systematic approaches to data collection (Whang et al., 2021), data-centric model training (Motamedi et al., 2021), benchmarking (Eyuboglu et al., 2022), and debugging (Rajani et al., 2022), as well as hubs that bring together resources and publications in the field (Data-centric AI, 2021).”