Matching Words and Pictures

By Kobus Barnard, Pinar Duygulu Sahin, D.A. Forsyth, Nando de Freitas, David M. Blei, Michael I. Jordan. 2003

1,816 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Other early work leveraged tags (but not natural language) associated with images for the task of semantic segmentation (Barnard et al., 2003).”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, methodology): “Several probabilistic models for representing joint multimodal probability distributions over images and sentences have been developed, using Deep Boltzmann Machines [16], log-bilinear models [17], and topic models [18, 19].”
Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, background): “Early work in this area includes generating single words or ﬁxed phrases from images (Duygulu et al., 2002; Barnard et al., 2003) or using contextual information to improve recognition (Gupta and Davis, 2008; Torralba et al., 2010).”
SUN database: Large-scale scene recognition from abbey to zoo (2010, background): “How many kinds of scenes are there? How can the knowledge about environmental scenes be organized? How do the current state-of-art scene models perform on more realistic and ill-controlled environments, and how does this compare to human performance? To date, computational work on scene and place re”
Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “The automatic generation of captions for images is a long standing and challenging problem in artiﬁcial intelligence [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19].”
RegionCLIP: Region-based Language-Image Pretraining (2021, background): “The most relevant work is learning from natural language, such as image tags [3, 8, 12, 25, 28] and text descriptions [11, 23, 43, 53, 61].”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “d annotations publicly available. 1 2. Related Work Dense image annotations. Our work shares the high-level goal of densely annotating the contents of images with many works before us. Barnard et al. [2] and Socher et al. [48] studied the multimodal correspondence between words and images to annotat”
Visual7W: Grounded Question Answering in Images (2015, background): “There have been years of effort in connecting the visual and textual information for joint learning [2, 19, 33, 35, 40, 52].”
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN) (2014, methodology): “…third category of methods, which is more related to our method, learns a probability density over the space of multimodal inputs (i.e. sentences and images), using for example, Deep Boltzmann Machines (Srivastava & Salakhutdinov (2012)), and topic models (Barnard et al. (2003); Jia et al. (2011)).”
Explain Images with Multimodal Recurrent Neural Networks (2014, methodology): “sentences and images), using for example, Deep Boltzmann Machines [35], and topic models [1, 15].”
DenseCap: Fully Convolutional Localization Networks for Dense Captioning (2015, background): “Several pioneering approaches have explored the task of describing images with natural language [1, 27, 12, 34, 42, 43, 28, 20].”
LabelMe: A Database and Web-Based Tool for Image Annotation (2008, methodology): “methods that can learn from such weakly labeled data include [ 2 ], which applies segmentation techniques to the Corel data, and [6], which applies interest point detectors to the Caltech data.”
A Joint Model of Language and Perception for Grounded Attribute Learning (2012, background): “In computer vision, the grounding problem often relates to detecting objects and attributes in visual information (e.g., see (Barnard et al., 2003)); however, these approaches primarily focus on isolated word meaning, rather than compositional semantic analyses.”
Objects that Sound (2017, background): “dence between the noun ‘dog’ and a spatial region in the image [7, 33].”
Jointly Modeling Embedding and Translation to Bridge Video and Language (2015, background): “In this direction, several works explore such probability distribution using topic models [3, 13] and neural networks [6, 16, 20, 31, 32, 33, 35].”
Joint Video and Text Parsing for Understanding Events and Answering Queries (2013, methodology): “In contrast to the bag-of-words representation widely used in the video understanding and multimedia literature (e.g., [1], [2], [3], [4]), in this work we propose to represent the joint interpretation of video and text in the form of a parse graph.”
SUN Database: Exploring a Large Collection of Scene Categories (2014, background): “Whereas most computational work on scene and place recognition has used a limited number of semantic categories, representing typical indoor and outdoor settings (Lazebnik et al. 2006; Fei-Fei and Perona 2005; Renninger and Malik 2004; Vogel and Schiele 2007; Oliva and Torralba 2001; Barnard et al. ”
Multimodal Distributional Semantics (2013, methodology): “The Battig benchmark was introduced by Baroni et al. (2010) and it is based on the Battig and Montague norms of Van Overschelde, Rawson, and Dunlosky (2004).”