A Step Toward More Inclusive People Annotations for Fairness

By Candice Schumann, Susanna Ricco, Utsav Prabhu, Vittorio Ferrari, Caroline Pantofaru. 2021

74 citations recorded by the source corpus.

The Open Images Dataset contains approximately 9 million images and is a widely accepted dataset for computer vision research. As is common practice for large datasets, the annotations are not exhaustive, with bounding boxes and attribute labels for only a subset of the classes in each image. In this paper, we present a new set of annotations on a subset of the Open Images dataset called the MIAP (More Inclusive Annotations for People) subset, containing bounding boxes and attributes for all of the people visible in those images. The attributes and labeling methodology for the MIAP subset were designed to enable research into model fairness. In addition, we analyze the original annotation methodology for the person class and its subclasses, discussing the resulting patterns in order to inform future annotation efforts. By considering both the original and exhaustive annotation sets, researchers can also now study how systematic patterns in training annotations affect modeling.

Subfields: Computer Vision and Pattern Recognition, Safety Research, Evaluation & Benchmarks, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Trevor Darrell
4. Ming-Hsuan Yang
5. Jitendra Malik
6. Xiaogang Wang
7. Andrew Zisserman
8. Cordelia Schmid

In Safety Research

1. Yi Tay
2. Carlos Riquelme
3. Hartwig Adam
4. Laura Gustafson

In Evaluation & Benchmarks

1. Neil Houlsby
2. Ross B. Girshick
3. Soravit Changpinyo
4. Alexander Kirillov
5. Weicheng Kuo
6. Radu Soricut
7. Xiaohua Zhai
8. Anelia Angelova

Citations in the corpus

• Segment Anything (2023, methodology): “We use the More Inclusive Annotations for People (MIAP) [87] dataset for gender presentation and age and a proprietary dataset for skin tone (see §C).”
• PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “analyze the perceived gender presentation of image subjects (Schumann et al., 2021) along with gender expressed through pronouns in text.”
• PaLI-X: On Scaling up a Multilingual Vision and Language Model (2023, methodology): “We present here an evaluation of how well PaLI-X performs across different subgroups using the MIAP [87] dataset.”
• VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, background): “While research has been conducted in the image generation and recognition domain [16, 54, 55, 83], this finding highlights the importance of continued research to develop strategies to mitigate issues and improve fairness for video generation.”