REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets

By Angelina Wang, Alexander Liu, Ryan Zhang, Anat Kleiman, Leslie Kim, Dora Zhao, Iroha Shirai, Arvind Narayanan, Olga Russakovsky. 2020

223 citations recorded by the source corpus.

Machine learning models are known to perpetuate and even amplify the biases present in the data. However, these data biases frequently do not become apparent until after the models are deployed. Our work tackles this issue and enables the preemptive analysis of large-scale datasets. REvealing VIsual biaSEs (REVISE) is a tool that assists in the investigation of a visual dataset, surfacing potential biases along three dimensions: (1) object-based, (2) person-based, and (3) geography-based. Object-based biases relate to the size, context, or diversity of the depicted objects. Person-based metrics focus on analyzing the portrayal of people within the dataset. Geography-based analyses consider the representation of different geographic locations. These three dimensions are deeply intertwined in how they interact to bias a dataset, and REVISE sheds light on this; the responsibility then lies with the user to consider the cultural and historical context, and to determine which of the revealed biases may be problematic. The tool further assists the user by suggesting actionable steps that may be taken to mitigate the revealed biases. Overall, the key aim of our work is to tackle the machine learning bias problem early in the pipeline. REVISE is available at https://github.com/princetonvisualai/revise-tool.

Subfields: Computer Vision and Pattern Recognition, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In AI Ethics, Fairness & Society

Citations in the corpus

Understanding and Evaluating Racial Biases in Image Captioning (2021, methodology): “REVISE [70] bias discovery.”
Multimodal datasets: misogyny, pornography, and malignant stereotypes (2021, methodology): “Various works [6, 7, 8, 9] have highlighted gender, racial, and geographical biases surrounding the sourcing of image datasets as well as the opacity of such endeavors [10].”
Data and its (dis)contents: A survey of dataset development and use in machine learning research (2020, methodology): “The REVISE tool by Wang et al.(80) can be used to identify unequal representation in image description datasets by leveraging features of the images and the corresponding texts.”
Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content? (2022, background): “…for language models, for instance, models producing stereotypical and derogatory content [Bender et al. 2021], and for vision model respectively CV datasets highlighting, e.g., gender and racial biases [Denton et al. 2021; Larrazabal et al. 2020; Steed and Caliskan 2021; Wang et al. 2020].”
Fair Attribute Classification through Latent Space De-biasing (2020, background): “Tools such as AI Fairness 360 [6] and REVISE [38] surface such biases in large-scale datasets and enable preemptive analysis.”
Advances, challenges and opportunities in creating data for trustworthy AI (2022, background): “Useful metadata includes statistics on the sex, gender, ethnicity and geographical location of the participants in the dataset, which helps to surface potential issues of underrepresented subpopulations 44 .”
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging (2023, methodology): “lighting the menorah) that are bound up with the identities of different social groups, again comparing them to some reference distribution (Shankar et al. 2017a; Wang et al. 2021; DeVries et al. 2019).”
Poisoning Web-Scale Training Datasets is Practical (2023, background): “Possibly most important among these is that uncurated data has serious implications for fairness, bias, and ethics [8, 50, 77].”
Quantifying Societal Bias Amplification in Image Captioning (2022, methodology): “In addition, REVISE [37] and CIFAR-10S [41] ease the task of identifying bias on datasets and models, respectively.”