Semantic Label Sharing for Learning with Many Categories

By Rob Fergus, Hector Bernal, Yair Weiss, Antonio Torralba. 2010

141 citations recorded by the source corpus.

In an object recognition scenario with tens of thousands of categories, even a small number of labels per category leads to a very large number of total labels required. We propose a simple method of label sharing between semantically similar categories. We leverage the WordNet hierarchy to define semantic distance between any two categories and use this semantic distance to share labels. Our approach can be used with any classifier. Experimental results on a range of datasets, upto 80 million images and 75,000 categories in size, show that despite the simplicity of the approach, it leads to significant improvements in performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Scalable multi-class object detection (2011, methodology): “Hierarchical taxonomies of object categories have been used previously for scalable image classification [16, 20, 10].”
Learning to share visual appearance for multiclass object detection (2011, background): “Most studies on transfer learning for object recognition have focused on multiclass recognition without a background class (saying if a crop image contains an object out of M possible classes [20, 14, 25, 11, 29]).”
Large-Scale Object Classification Using Label Relation Graphs (2014, methodology): “In particular, label hierarchies have been used to share representations [15,2,8,17] and combine models [18,38,27].”
Hierarchical semantic indexing for large scale image retrieval (2011, background): “…[4, 29, 7],and hints at the promise of exploiting such structure for clas- 1This is actually the case in most work on similarity learning [33, 15, 6]. sification when evaluated in terms of the hierarchy [7] as well as showing improved classification given very small amounts of training data [13].”
Learning semantic relationships for better action retrieval in images (2015, background): “Semantic hierarchy for vision In the last few years, different works [1, 7, 8, 9, 11, 16, 24, 26, 37, 44, 47] have tried to use preexisting structure between labels to train better models for image classification, and object segmentation [20].”
Decoupling Zero-Shot Semantic Segmentation (2021, background): “The key idea in zero-shot learning is to transfer kowledge from seen classes to unseen classes via semantic representation, such as the semantic attributes [2,24,28,30], concept ontology [16,37,39,47,48] and semantic word vectors [17, 41, 60].”
Hedging your bets: Optimizing accuracy-specificity trade-offs in large scale visual recognition (2012, background): “Our work is also inspired by an emerging trend in computer vision studying large scale visual recognition [19, 27, 13, 12, 31, 7, 26, 24].”
Zero-Shot Recognition via Semantic Embeddings and Knowledge Graphs (2018, background): “Researchers have proposed several approaches on how to use knowledge graphs for object recognition [12, 43, 30, 35, 39, 9, 8, 29, 49, 47, 26].”
Learning multiple visual tasks while discovering their structure (2015, methodology): “Empirical tests show that the proposed method compares favorably to state of the art techniques and further allows to recover interpretable structures, a problem of interest in its own right.”
Fine-Grained Image Classification by Exploring Bipartite-Graph Labels (2015, background): “Previous works have shown the benefits of leveraging the semantic class hierarchy using either unsupervised [2, 50] or supervised [11, 18, 39] methods.”
Semantic Kernel Forests from Multiple Taxonomies (2012, methodology): “We demonstrate our method on challenging object recognition datasets, and show that interleaving multiple taxonomic views yields significant accuracy improvements.”
Birdlets: Subordinate categorization using volumetric primitives and pose-normalized appearance (2011, methodology): “Supervised methods that utilize such information (as contained in WordNet for example) include the sharing of training examples across semantically similar categories [26] and combining information from different levels of the semantic hierarchy [50].”
Human Uncertainty Makes Classification More Robust (2019, background): “The label hierarchies used to aid recognition can be manually constructed [6, 3], derived from linguistic knowledge bases [10, 9], or learned automatically [14, 19].”
K-LITE: Learning Transferable Visual Models with External Knowledge (2022, background): “…to associate observed and non-observed classes through some form of auxiliary information, which can be either implicit such as pre-trained semantic embeddings [92, 78, 10], or explicit such as attributes [21, 43, 35], text [18, 19, 72, 70], knowledge graphs [91, 74], or rules and ontologies [23].”
Write a Classifier: Zero-Shot Learning Using Purely Textual Descriptions (2013, background): “We denote the number of classes available at training as Nsc, where sc indicates “seen classes”.”
Which Looks Like Which: Exploring Inter-class Relationships in Fine-Grained Visual Categorization (2014, unknown): “For example, adopting class taxonomies or hierarchies [6,23] may help train better prediction models by sharing appearance [20,30], visual parts [33], or classifiers [4].”
Convex Learning of Multiple Tasks and their Structure (2015, background): “…, · ) Ac i , the squared norm can be written as f 2 H = ni,j k ( x i , x j ) c ⊤ i Ac j = tr ( AC ⊤ KC ) where C ∈ R n × T is the matrix with i -th row corresponding to the co-efﬁcient vector c i ∈ R T of f . the Notice that we have re-ordered the index i to be in { 1 , . . . , n to ease notation.”
Relative attributes (2011, background): “Recent work on largescale recognition exploits WordNet-based information to specify a semantic-distance sensitive classifier [18], or to make do with few labels by sharing training images among semantically similar classes [17].”