Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels

By Ishan Misra, C. Lawrence Zitnick, Margaret Mitchell, Ross B. Girshick. 2015

236 citations recorded by the source corpus.

When human annotators are given a choice about what to label in an image, they apply their own subjective judgments on what to ignore and what to mention. We refer to these noisy "human-centric" annotations as exhibiting human reporting bias. Examples of such annotations include image tags and keywords found on photo sharing sites, or in datasets containing image captions. In this paper, we use these noisy annotations for learning visually correct image classifiers. Such annotations do not use consistent vocabulary, and miss a significant amount of the information present in an image, however, we demonstrate that the noise in these annotations exhibits structure and can be modeled. We propose an algorithm to decouple the human reporting bias from the correct visually grounded labels. Our results are highly interpretable for reporting "what's in the image" versus "what's worth saying." We demonstrate the algorithm's efficacy along a variety of metrics and datasets, including MS COCO and Yahoo Flickr 100M.We show significant improvements over traditional algorithms for both image classification and image captioning, doubling the performance of existing methods in some cases.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, background): “…most of the existing MRC datasets, all queries and passages in Re Co RD are automatically mined from news articles, which maximally reduces the human elicitation bias (Gordon and Van Durme, 2013; Misra et al., 2016; Zhang et al., 2017), and the data collection method we propose is cost-efﬁcient.”
Exposing and Correcting the Gender Bias in Image Captioning Datasets and Models (2019, background): “[Misra et al., 2016] discuss the reporting bias in image descriptions but do not highlight gender bias.”
Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints (2017, background): “Data collection biases have been discussed in the context of creating image corpus (Misra et al., 2016; van Miltenburg, 2016) and text corpus (Gordon and Van Durme, 2013; Van Durme, 2010).”
Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods (2018, background): “Various other approaches have been proposed to produce “fair” classifiers (Calders et al., 2009; Feldman et al., 2015; Misra et al., 2016).”
Deep Clustering for Unsupervised Learning of Visual Features (2018, background): “Replacing labels by raw metadata leads to biases in the visual representations with unpredictable consequences [41].”
Mitigating Gender Bias in Natural Language Processing: Literature Review (2019, background): “…to NLP; other ﬁelds in computer science such as data mining, machine learning, and security also study gender bias (Calders and Verwer, 2010; Feldman et al., 2015; Hardt et al., 2016; Misra et al., 2016; Kleinberg et al., 2016; Pleiss et al., 2017; Beutel et al., 2017; Kilbertus et al., 2017).”
The More You Know: Using Knowledge Graphs for Image Classification (2016, methodology): “We train and test in the multi-label setting [24], and evaluate on the minival set [28].”
Ordinal Common-sense Inference (2016, background): “5 4 For further background see discussions by Van Durme (2010), Gordon and Van Durme (2013), Rudinger et al. (2015) and Misra et al. (2016).”
Learning from Noisy Large-Scale Datasets with Minimal Supervision (2017, background): “[22] who model noise arising from missing, but visually present labels.”
Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias (2018, methodology): “This means that the structure of noise can be modelled as a latent variable and decoupled during training [18].”
Talking about other people: an endless range of possibilities (2018, background): “Earlier work has looked at stereotyping behavior, reporting bias, and the use of negations in image descriptions (van Miltenburg, 2016; Misra et al., 2016; van Miltenburg et al., 2016, 2017), and recently Van Miltenburg et al. (2018) provided an overview of measures to quantify diversity.”
Deconfounded Image Captioning: A Causal Retrospect (2020, background): “However, since humans are living in a world ﬁlled with various biases, e.g., long-tailed concept distributions [4], [5], reporting bias [6], [7], and language bias [8], [9], we are easily trapped in the loop: “making a dataset”–“it’s biased”–“making a new one”.”
Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (2022, background): “human reporting bias When users rely on automation as a heuristic replacement for their own information seeking and processing [272].”