Learning the semantics of words and pictures

By Kobus Barnard, D.A. Forsyth. 2001

620 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Molecular Biology, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Molecular Biology

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “The automatic generation of captions for images is a long standing and challenging problem in artiﬁcial intelligence [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19].”
A Multi-View Embedding Space for Modeling Internet Images, Tags, and Their Semantics (2012, background): “Some of the earliest research on images and text (Barnard and Forsyth, 2001; Blei and Jordan, 2003; Blei et al., 2003; Duygulu et al., 2002; Lavrenko et al., 2003) has focused on learning the co-occurrences between image regions and tags using a generative model.”
Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels (2015, background): “It can impact the training process of models and decrease their predictive accuracy [1, 21, 37].”
Reinforcement Learning for Mapping Instructions to Actions (2009, methodology): “Examples include configuring software based on how-to guides and operating simulators using instruction manuals.”
Learning to Win by Reading Manuals in a Monte-Carlo Framework (2011, methodology): “Examples of such approaches are the methods of Barnard and Forsyth (2001), and Liang et al.”
Discovering objects and their location in images (2005, background): “This may range from specifying the object’s location and segmentation, as in face detection [17, 23], to providing only auxiliary data indicating the object’s identity [1, 2, 8, 24].”
Deep Convolutional Ranking for Multilabel Image Annotation (2013, methodology): “Early work in this area was mostly devoted to annotation models inspired by machine translation techniques [1, 8].”
Beyond Nouns: Exploiting Prepositions and Comparative Adjectives for Learning Visual Classifiers (2008, methodology): “al [6] presented a generative model for image annotation that induces hierarchical structure from the co-occurrence data.”
Some Objects Are More Equal Than Others: Measuring and Predicting Importance (2008, background): “What are the next steps toward image understanding? A full description of complex scenes, currently appears to be out of reach (although there is interesting work in that direction [8]).”
Measuring and Predicting Importance of Objects in Our Visual World (2007, neutral): “How should such a list be produced?”
A Statistical Model for General Contextual Object Recognition (2004, background): “Previous work shows that it is reasonable to use such loosely labeled data for problems in vision and image retrieval[1, 4, 13, 11, 2, 7].”
CLUE: cluster-based retrieval of images by unsupervised learning (2005, methodology): “[2] presented a probability-based similarity measure that combines the information provided by text and the visual information provided by image features.”
Measuring Machine Intelligence Through Visual Question Answering (2016, background): “Are tasks such as image captioning [3, 22, 27, 16, 19, 15, 6, 11, 25, 21, 20, 32] promising candidates for testing artiﬁcial intelligence?”
Supervised Learning of Semantic Classes for Image Annotation and Retrieval (2007, methodology): “The basic idea underlying the unsupervised learning formulation [3], [4], [12], [13], [15], [21], [31] is to introduce a variable L that encodes hidden states of the world.”
Automatic Caption Generation for News Images (2013, methodology): “Normalized cuts (Shi and Malik, 2000) are widely used to segment images as they produce acceptable output for a wide range of occasions and non-surprisingly are popular in image annotation research (Barnard and Forsyth, 2001; Duygulu et al., 2002; Barnard et al., 2002; Lavrenko et al., 2003; Blei an”
Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary (2002, methodology): “There exist some methods that cluster image representations and text to produce a representation of a joint distribution linking images and words [1, 2].”
Large-Scale Supervised Multimodal Hashing with Semantic Correlation Maximization (2014, background): “As the development of information technology, more and more multimodal data have been available in many applications, especially in multimedia domains (Barnard and Forsyth 2001; Chua et al. 2009; Song et al. 2011; Chen et al. 2012; Wu et al. 2014).”
Peekaboom: a game for locating objects in images (2006, methodology): “[4,5,9,10]) rely on machine learning: train an algorithm to perform a visual task by showing it example images in which the task has already been performed.”