Collecting Image Annotations Using Amazon’s Mechanical Turk

By Cyrus Rashtchian, Peter Young, Micah Hodosh, Julia Hockenmaier. 2010

753 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Microsoft COCO: Common Objects in Context (2014, methodology): “points on the object [47]. Finally, our dataset could provide a good benchmark for other types of labels, including scene types [3], attributes [8], [7] and perhaps full sentence written descriptions [48]. APPENDIX OVERVIEW In the appendix, we provide 1) detailed descriptions of the AMT user interfa”
Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Modern work on image-text retrieval has relied on a set of crowd-sourced sentence level image caption evaluation datasets like Pascal1K (Rashtchian et al., 2010), Flickr8K (Hodosh et al.”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, methodology): “In particular, we report dramatic improvements over state of the art methods on image-sentence retrieval tasks on Pascal1K [2], Flickr8K [3] and Flickr30K [4] datasets.”
Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, methodology): “Then we analyze Image Search with Query Sentences : to query each model with a sentence in order to ﬁnd an image showing that sen-213 Figure 5: Examples from the dataset of images and their sentence descriptions (Rashtchian et al., 2010).”
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (2013, methodology): “Our annotation process and quality control are described in detail in Rashtchian et al. (2010)'s paper.”
Show and tell: A neural image caption generator (2014, methodology): “Descriptions were preprocessed with basic tokenization, keeping all words that appeared at least 5 times in the training set.”
CIDEr: Consensus-based image description evaluation (2014, methodology): “The most popular dataset is the UIUC Pascal Sentence Dataset [35].”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, methodology): “3.3 Quality Control Since worker quality on AMT is highly variable (Sorokin and Forsyth 2008; Rashtchian et al. 2010), we take a combination of measures to ensure the integrity of annotations.”
SPICE: Semantic Propositional Image Caption Evaluation (2016, methodology): “To create the PASCAL-50S dataset [12], 1,000 images from the UIUC PASCAL Sentence Dataset [37]—originally containing ﬁve captions per image—were annotated with 50 captions each using AMT.”
Multimodal datasets: misogyny, pornography, and malignant stereotypes (2021, background): “The 2010-2020 decade saw the emergence of smaller scale initiatives such as the UIUC-Pascal-Sentence Dataset [46], Microsoft COCO [47]: Common Objects in Context dataset (330,000 images with 5 independent human generated captions), the Yahoo Flickr Creative Commons 100 Million (YFCC100M) Dataset…”
Explain Images with Multimodal Recurrent Neural Networks (2014, methodology): “In the experiments, we validate our model on three benchmark datasets: IAPR TC-12 [9], Flickr 8K [31], Flickr 30K [14] and MS COCO [23].”
Visual Madlibs: Fill in the blank Image Generation and Question Answering (2015, methodology): “The MS COCO dataset [21] collects general image descriptions following a similar methodology to previous efforts for collecting general image descriptions, e.g. [28, 34].”
SemEval-2014 Task 10: Multilingual Semantic Textual Similarity (2014, methodology): “The Images data set is a subset of the PAS-CAL VOC-2008 data set (Rashtchian et al., 2010), which consists of 1,000 images and has been used by a number of image description systems.”
Learning a Recurrent Visual Representation for Image Caption Generation (2014, methodology): “For evaluation we perform experiments on several standard datasets that are used for sentence generation and the sentence-image retrieval task: PASCAL 1K [31] The dataset contains a subset of images from the PASCAL VOC challenge.”
TGIF: A New Dataset and Benchmark on Animated GIF Description (2016, neutral): “[28] and Chen et al .”
Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge (2016, neutral): “The most obvious case for transfer learning and data size is between Flickr30k and Flickr8k.”
Stereotyping and Bias in the Flickr30K Dataset (2016, background): “Rashtchian et al. (2010) do note that some descriptions are speculative in nature, which they say hurts the accuracy and the consistency of the descriptions.”