Learning Models for Object Recognition from Natural Language Descriptions

By Josiah Wang, Katja Markert, Mark Everingham. 2009

237 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Ecological Modeling, Genetics, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Ecological Modeling

In Genetics

In Natural Language Processing

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “However, as mentioned in the introduction and approach section, methods of leveraging natural language descriptions in computer vision well predate the use of this specific term, especially for image retrieval (Mori et al., 1999) and object classification (Wang et al., 2009).”
RegionCLIP: Region-based Language-Image Pretraining (2021, background): “The most relevant work is learning visual representations from natural language, such as image tags [3, 8, 12, 25, 28] and text descriptions [11,23,43,53,62].”
Inferring the Why in Images (2014, background): “Language in Vision: The community has been incorporating natural language into computer vision over the last decade to great success, from generating sentences from images [19], producing visual models from sentences [36, 33], and aiding in contextual models [26, 21] to name a few.”
Crowdsourcing Feature Discovery via Adaptively Chosen Comparisons (2015, background): “As mentioned, a common approach for crowdsourcing named features is image tagging, see, e.g., the ESP game [13].”
F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models (2022, background): “…the promise of representation learning from natural language associated with images, such as image tags (Chen & Gupta, 2015; Divvala et al., 2014; Joulin et al., 2016) or text descriptions (Desai & Johnson, 2021; He & Peng, 2017; Sariyildiz et al., 2020; Wang et al., 2009; Zhong et al., 2021).”
Automatic Caption Generation for News Images (2013, background): “Examples include associating names mentioned in the captions to faces depicted in news images (e.g., [23], [24]), verbs to body poses [25], and learning models for recognizing objects [26] and their relative importance [27].”
Attribute-Based Detection of Unfamiliar Classes with Humans in the Loop (2013, background): “The emerging trend of computer vision entering the wild in the form of semi-automated field guides (Visipedia [28], Leafsnap [17], butterflies [39], etc.”
How Do You Tell a Blackbird from a Crow? (2013, background): “, trees [13], flowers [16], butterflies [8, 27], dogs [15, 18, 19], and birds [1, 6, 8, 9, 29, 30].”
Understanding Objects in Detail with Fine-Grained Attributes (2014, background): “Attributes can also be used to transfer expert knowledge to a computer vision system in zero-shot learning [25] [43], in finding relations between object categories [34] [42], in incorporating annotator's rationales and other forms of feedback [8] [35].”
Predicting Motivations of Actions by Leveraging Text (2014, unknown): “Language in Vision: The community has recently been incorporating natural language into computer vision, such as generating sentences from images [20, 15, 36], producing visual models from sentences [44, 38], and aiding in contextual models [26, 22].”
Relative attributes (2011, background): “For example, researchers have shown their impact for strengthening facial verification [5], object recognition [6, 8, 16], generating descriptions of unfamiliar objects [1], and to facilitate “zero-shot” transfer learning [2], where one trains a classifier for an unseen object simply by specifying w”
Sharing features between objects and their attributes (2011, background): “Usually attribute-object associations are manually specified, but somework explores ways to obtain them automatically [26, 6, 21].”
Discovering localized attributes for fine-grained recognition (2012, background): “We ﬁnd on these datasets that our discovered local attributes outperform those generated by human experts and by other strong baselines, on ﬁne-grained image classiﬁcation tasks.”
Baby talk: Understanding and generating simple image descriptions (2011, background): “Most past approaches have either constructed the set of attribute terms in an ad hoc manner or taken them from an application appropriate ontology [13], [19], [27], [29], but some approaches have tried to learn the attributes directly from image-text pairs [6], [43].”
Attribute-Based Classification for Zero-Shot Visual Object Categorization (background): “It has also been shown that new attributes can be found by text mining [38], [39], [40], and that object classes themselves can act as attributes for other tasks [41].”
BabyTalk: Understanding and Generating Simple Image Descriptions (2013, background): “Most past approaches have either constructed the set of attribute terms in an ad hoc manner or taken them from an application appropriate ontology [13], [19], [27], [29], but some approaches have tried to learn the attributes directly from image-text pairs [6], [43].”
Automatic Attribute Discovery and Characterization from Noisy Web Data (2010, background): “Recent work moves toward automating part of the attribute learning process, but is focused on the constrained setting of butterfly field guides and uses hand coded visual features specific to that setting, language templates, and predefined attribute lists (lists of color terms etc) [11] to obtain v”
Start from Scratch: Towards Automatically Identifying, Modeling, and Naming Visual Attributes (2014, methodology): “There are also automatic attribute discovery and naming methods that fall into specific domains like butterfly [37] and fashion productsbengio2009learning recogntion [3].”