No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World

By S. Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, David Sculley. 2017

343 citations recorded by the source corpus.

Modern machine learning systems such as image classifiers rely heavily on large scale data sets for training. Such data sets are costly to create, thus in practice a small number of freely available, open source data sets are widely used. We suggest that examining the geo-diversity of open data sets is critical before adopting a data set for use cases in the developing world. We analyze two large, publicly available image data sets to assess geo-diversity and find that these data sets appear to exhibit an observable amerocentric and eurocentric representation bias. Further, we analyze classifiers trained on these data sets to assess the impact of these training distributions and find strong differences in the relative performance on images from different locales. These results emphasize the need to ensure geo-representation when constructing data sets for use in the developing world.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Media Technology, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Media Technology

In Evaluation & Benchmarks

Citations in the corpus

AI and the Everything in the Whole Wide World Benchmark (2021, methodology): “2% of images are from China and India respectively, despite the fact that those countries are the most populous countries on the planet [Shankar et al., 2017].”
Re-imagining Algorithmic Fairness in India and Beyond (2021, background): “For instance, [190] highlights the glaring gaps in geo-diversity of open datasets such as ImageNet and Open Images that drive much of the computer”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Data availability differs due to geographic biases in data collection [216], disparate digitization of content globally due to varying levels of internet access for digitizing content, and infrastructure created to support some languages or accents over others, among other reasons.”
FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age (2019, background): “Studies in algorithmic fairness have focused on either 1) discovering (auditing) existing bias in datasets or systems [50, 7, 30], 2) making a better dataset [39, 1], or 3) designing a better algorithm or model [12, 1, 47, 71, 70].”
Queens Are Powerful Too: Mitigating Gender Bias in Dialogue Generation (2019, background): “However, they can be a starting point for bias (Shankar et al., 2017; Clark et al., 2019; Henderson et al., 2018), as bias in the personas propagates to subsequent conversations.”
Does Object Recognition Work for Everyone? (2019, background): “In line with prior analyses [38], the density maps demonstrate that the computer-vision dataset severely undersample visual scenes in a range of geographical regions with large populations, in particular, in Africa, India, China, and South-East Asia.”
Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development (2021, neutral): “Object recognition datasets have a documented bias towards Western countries [27, 105].”
A Survey on Bias and Fairness in Machine Learning (2019, background): “In [116], researchers studied and showed that these datasets suffer from representation bias and advocate for the need to incorporate geo-diversity and inclusion while creating such datasets.”
Federated Learning with Matched Averaging (2020, positive): “It has been shown that an observable amerocentric and eurocentric bias is present in the widely used ImageNet dataset (Shankar et al., 2017; Russakovsky et al., 2015).”
Large image datasets: A pyrrhic win for computer vision? (2020, background): “The Inclusive images challenge [73], for example, already incorporated blurring during dataset curation 14 and addressed the downstream effects surrounding change in predictive power of the models trained on the blurred versions of the dataset curated.”
What are the Biases in My Word Embedding? (2018, background): “Finally, while a significant portion of the empirical research on algorithmic fairness has focused on the societal biases that are most pressing in the countries where the majority of researchers currently conducting the work are based, the literature also contains examples of biases that may be of ”
On the genealogy of machine learning datasets: A critical history of ImageNet (2021, background): “Worse, data collection efforts aimed at increasing the representation of margina-lized groups within training data are often executed through exploitative or extractive mechanisms (Chutel, 2018; Solon, 2019).”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, positive): “…et al., 2020): for example, publicly available image datasets overrepresent images from the United States and Europe, producing drops in performance on images from the developing world (Shankar et al., 2017) and prompting creation of more geographically diverse image datasets (Atwood et al., 2020).”
A Study of Face Obfuscation in ImageNet (2021, background): “For ImageNet, researchers have examined and attempted to mitigate issues such as geographic diversity, the category vocabulary, and imbalanced representation (Shankar et al., 2017; Stock & Cisse, 2018; Dulhanty & Wong, 2019; Yang et al., 2020).”
Characterising Bias in Compressed Models (2020, background): “…within facial analysis datasets, object detection datasets tend to under-represent images from lower income and non-Western regions (Shankar et al., 2017; DeVries et al., 2019), activity recognition datasets exhibit stereotype-aligned gender biases (Zhao et al., 2017), and word…”