Stereotyping and Bias in the Flickr30K Dataset

By Emiel van Miltenburg. 2016

98 citations recorded by the source corpus.

An untested assumption behind the crowdsourced descriptions of the images in the Flickr30K dataset (Young et al., 2014) is that they "focus only on the information that can be obtained from the image alone" (Hodosh et al., 2013, p. 859). This paper presents some evidence against this assumption, and provides a list of biases and unwarranted inferences that can be found in the Flickr30K dataset. Finally, it considers methods to find examples of these, and discusses how we should deal with stereotype-driven descriptions in future applications.

Subfields: Biophysics, Computer Vision and Pattern Recognition, Molecular Biology, Radiology, Nuclear Medicine and Imaging, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

In Molecular Biology

In Radiology, Nuclear Medicine and Imaging

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

Gender Bias in Coreference Resolution (2018, background): “Prior work also analyzes social and gender stereotyping in existing NLP and vision datasets (van Miltenburg, 2016; Rudinger et al., 2017).”
Exposing and Correcting the Gender Bias in Image Captioning Datasets and Models (2019, background): “[van Miltenburg, 2016] discuss the bias in Flickr30k Dataset and touches on gender-related issues in the captions.”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, background): “Moreover, it is generally feasible to have only a limited number of users to annotate training images, which may lead to a biased model [17, 50, 59].”
Understanding and Evaluating Racial Biases in Image Captioning (2021, background): “Prior works [54, 65] show that people are more likely to use racial descriptors when describing non-white individuals.”
Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints (2017, background): “Data collection biases have been discussed in the context of creating image corpus (Misra et al., 2016; van Miltenburg, 2016) and text corpus (Gordon and Van Durme, 2013; Van Durme, 2010).”
Data and its (dis)contents: A survey of dataset development and use in machine learning research (2020, background): “…relating to race, gender, (dis)ability, and more [Garg et al., 2018, Hutchinson et al., 2020] and correlations between gender and activities depicted in computer vision datasets have been shown to reﬂect common gender stereotypes [Zhao et al., 2017, Burns et al., 2018, van Miltenburg, 2016].”
Talking about other people: an endless range of possibilities (2018, methodology): “We created a taxonomy of the labels used to refer to other people by manually sorting the entity labels into different semantic categories, instead of clustering the labels (as in Van Miltenburg 2016).”
Algorithmic Fairness (2020, background): “Second, the labels of the data usually depend on annotators and are not always accurate [129, 134].”
Discovering and Categorising Language Biases in Reddit (2020, background): “Stereo-types can be understood as ideas about how (groups of) people commonly behave (van Miltenburg 2016).”
Implicit Diversity in Image Summarization (2019, background): “Van Miltenburg [80] provided evidence of stereotype-bias in a popular dataset of Flickr images annotated with crowdsourced descriptions.”
A Review on Fairness in Machine Learning (2022, background): “For example, Stock and Cisse [193] show that human annotators fail to capture parts of the visual concepts in images, and Van Miltenburg [202] show that the crowdsourced descriptions of the images in the Flickr30K dataset are often based on stereotypes and prejudices of the annotators; another chall”
Measuring the Diversity of Automatic Image Descriptions (2018, background): “It is unclear which is meant by Ferraro et al., but it would be interesting to see whether machine-generated descriptions are more closely tied to perception than human descriptions, who also speculate about the context of the images (van Miltenburg, 2016).”