Beyond Nouns: Exploiting Prepositions and Comparative Adjectives for Learning Visual Classifiers

By Abhinav Kumar Gupta, Larry S. Davis. 2008

201 citations recorded by the source corpus.

Learning visual classifiers for object recognition from weakly labeled data requires determining correspondence between image regions and semantic object classes. Most approaches use co-occurrence of "nouns" and image features over large datasets to determine the correspondence, but many correspondence ambiguities remain. We further constrain the correspondence problem by exploiting additional language constructs to improve the learning process from weakly labeled data. We consider both "prepositions" and "comparative adjectives" which are used to express relationships between objects. If the models of such relationships can be determined, they help resolve correspondence ambiguities. However, learning models of these relationships requires solving the correspondence problem. We simultaneously learn the visual features defining "nouns" and the differential visual features defining such "binary-relationships" using an EM-based approach.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, background): “The field has tackled a diverse set of tasks such as captioning [15, 13, 40, 39, 24], alignment [13, 17, 38], Q&A [22, 21], visual model learning from textual descriptions [9, 29], and semantic visual search with natural multisentence queries [19].”
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations (2016, methodology): “The scene graph representation has been shown to improve semantic image retrieval (Johnson et al., 2015, Schuster et al., 2015) and image captioning (Farhadi et al., 2009, Chang et al., 2014, Gupta and Davis, 2008).”
Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, methodology): “Early work in this area includes generating single words or ﬁxed phrases from images (Duygulu et al., 2002; Barnard et al., 2003) or using contextual information to improve recognition (Gupta and Davis, 2008; Torralba et al., 2010).”
CIDEr: Consensus-based image description evaluation (2014, background): “Other works have modeled prepositional relationships [16], attributes (adjectives) [23, 34], and visual phrases (i.”
Scene Graph Generation by Iterative Message Passing (2017, background): “One of the most popular ways of representing a visual scene is through text descriptions [14, 34, 44].”
Visual Relationship Detection with Language Priors (2016, neutral): “Relationships have also improved object localization [32,33,6,34].”
Weakly-Supervised Learning of Visual Relations (2017, background): “Triplet learning has been addressed in various tasks such as mining typical relations (knowledge extraction) [7, 43, 52, 54], reasoning [21, 35, 45], object detection [17, 44], image retrieval [23] or fact retrieval [11].”
Don't just listen, use your imagination: Leveraging visual common sense for non-visual tasks (2015, background): “Approaches that automatically produce a textual description of images [20, 13, 27] or synthe-size scenes corresponding to input textual descriptions [46] can beneﬁt from reasoning about all these different “W” questions and other high-level information.”
An empirical study of context in object detection (2009, background): “It also may indicate the presence and location (spatial context) of other objects and materials [10, 11, 12, 37].”
Linking People in Videos with "Their" Names Using Coreference Resolution (2014, background): “Another line of work focuses on learning classifiers from images with text [5, 15, 21].”
Every Picture Tells a Story: Generating Sentences from Images (2010, unknown): “Such relations are revealing: Gupta and Davis show that respecting likely spatial relations between objects markedly improves the accuracy of both annotation and placing [7].”
LabelMe: Online Image Annotation and Applications (2010, methodology): “Finally, there are oth er recent relevant methods to recover geometric information f or images [21], [58], [9], [46], [67], [33], [19], [38], [83].”
Baby talk: Understanding and generating simple image descriptions (2011, methodology): “These relationships were used as contextual models to improve labeling accuracy, but the spatial relationships themselves were not considered outputs in their own right [10], [20], [38], [41].”
Beyond Actions: Discriminative Models for Contextual Group Activities (2010, unknown): “For example, work has been done on exploiting contextual information between scenes and objects [30], objects and objects [8, 33, 13, 14], objects and so-called “stuff” (amorphous spatial extent, e.”
Actions in context (2009, background): “A related approach deriving object relations from textual annotations in still images has been recently presented in [9].”
Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora (2010, methodology): “Both types of words are mapped to this space by kernelized Canonical Correlation Analysis, making this the first usage of kCCA in segmentation.”
Situation Recognition: Visual Semantic Role Labeling for Image Understanding (2016, background): “These intuitions are often built directly into datasets by framing activity recognition as a discrete classification problem, with a small set of multi-word category labels that combine a verb with a scene or object [4, 10, 22, 44, 48, 49].”
Moving vistas: Exploiting motion for describing scenes (2010, background): “The large intra-class variation due to unconstrained settings and the complex underlying physics present challenging problems in modeling scene dynamics.”