Fine-Grained Image Classification via Combining Vision and Language

By Xiangteng He, Yuxin Peng. 2017

184 citations recorded by the source corpus.

Fine-grained image classification is a challenging task due to the large intra-class variance and small inter-class variance, aiming at recognizing hundreds of sub-categories belonging to the same basic-level category. Most existing fine-grained image classification methods generally learn part detection models to obtain the semantic parts for better classification accuracy. Despite achieving promising results, these methods mainly have two limitations: (1) not all the parts which obtained through the part detection models are beneficial and indispensable for classification, and (2) fine-grained image classification requires more detailed visual descriptions which could not be provided by the part locations or attribute annotations. For addressing the above two limitations, this paper proposes the two-stream model combing vision and language (CVL) for learning latent semantic representations. The vision stream learns deep representations from the original visual information via deep convolutional neural network. The language stream utilizes the natural language descriptions which could point out the discriminative parts or characteristics for each image, and provides a flexible and compact way of encoding the salient visual aspects for distinguishing sub-categories. Since the two streams are complementary, combing the two streams can further achieves better classification accuracy. Comparing with 12 state-of-the-art methods on the widely used CUB-200-2011 dataset for fine-grained image classification, the experimental results demonstrate our CVL approach achieves the best performance.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “More recently, He & Peng (2017) and (Liang et al., 2020) demonstrated using natural language descriptions and explanations to improve fine-grained visual classification of birds.”
ALICE: Active Learning with Contrastive Natural Language Explanations (2020, background): “(2020); He and Peng (2017) encode natural language explanations as additional features to assist classification.”
Shaping Visual Representations with Language for Few-Shot Classification (2019, neutral): “Language has been shown to assist visual classiﬁcation in various settings, including traditional visual classiﬁcation with no transfer [16] and with language available at test time in the form of class labels or descriptions for zero-[10, 11, 27] or few-shot [24, 33] learning.”
RegionCLIP: Region-based Language-Image Pretraining (2021, background): “The most relevant work is learning visual representations from natural language, such as image tags [3, 8, 12, 25, 28] and text descriptions [11,23,43,53,62].”
Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning (2018, background): “Beyond pixels, we also leverage other information including parts [66, 7, 67], attributes [57, 18], human interactions [8, 13] and text descriptions [42, 22].”
Fine-Grained Representation Learning and Recognition by Exploiting Hierarchical Semantic Embedding (2018, methodology): “Here, we evaluate on recognizing the categories of the finest level Bilinear-CNN (B-CNN) [25], Compact Bilinear CNN (CB-CNN) [9], Two-Level Attention Network (TLAN) [43], Diverse Attention Network(DAN)[49],FullyConvolutionalAttentionalNetwork(FCAN)[28], RecurrentAttention(RA-CNN)[8],CombineVisionand”
Zero-Shot Learning from scratch (ZFS): leveraging local compositional representations (2020, background): “…this approach appears to be well-justiﬁed, as state-of-the-art results on various ZSL (Yosinski et al., 2014; Sun et al., 2017; Huh et al., 2016; Azizpour et al., 2015) and non-ZSL benchmarks (Li et al., 2019; Zhang et al., 2018b; He & Peng, 2017) all learn on top of these pretrained backbones.”
Locality and compositionality in zero-shot learning (2019, background): “…this approach appears to be well-justiﬁed, as state-of-the-art results on various ZSL (Yosinski et al., 2014; Sun et al., 2017; Huh et al., 2016; Azizpour et al., 2015) and non-ZSL benchmarks (Li et al., 2019; Zhang et al., 2018b; He & Peng, 2017) all learn on top of similar pretrained backbones.”
Audio Visual Attribute Discovery for Fine-Grained Object Recognition (2018, methodology): “Comparing with (He and Peng 2017) which uses the text descriptions as the context, our approach obtains a litter improvements.”
MotionCLIP: Exposing Human Motion Generation to CLIP Space (2022, background): “Neural networks have successfully learned powerful latent representations coupling natural images with natural language describing it [He and Peng 2017; Ramesh et al. 2021].”
Text-to-image Synthesis via Symmetrical Distillation Networks (2018, unknown): “We can find many successful applications of this paradigm in the various state-of-the-art methods, such as object detection [7], semantic segmentation [21] and fine-grained image classification [13].”
WildFish: A Large Benchmark for Fish Recognition in the Wild (2018, background): “embedding for multi-modal retrieval [7, 31, 41], zero/few-shot learning [2, 13, 30, 39], fine-grained recognition [2, 19, 31], and so on.”
Fine-Grained Image Analysis With Deep Learning: A Survey (2021, background): “Subsequently, Peng et al. [85] and Zheng et al. [88] proposed multi-level attention models to obtain hierarchical attention information (i.e., both object-and part-level).”
FineGAN: Unsupervised Hierarchical Disentanglement for Fine-Grained Object Generation and Discovery (2018, methodology): “Some methods require additional part [58, 6, 55], attribute [14], or text [39, 19] annotations.”
Knowledge Concentration: Learning 100K Object Classifiers in a Single CNN (2017, background): “[9] proposed a two-stream model that combines the vision and language for learning latent semantic representations for fine-grained classification.”
What does a platypus look like? Generating customized prompts for zero-shot image classification (2022, methodology): “[45] and [19] use natu-ral language descriptions of bird types to train a multimodal classiﬁcation model.”
F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models (2022, background): “…the promise of representation learning from natural language associated with images, such as image tags (Chen & Gupta, 2015; Divvala et al., 2014; Joulin et al., 2016) or text descriptions (Desai & Johnson, 2021; He & Peng, 2017; Sariyildiz et al., 2020; Wang et al., 2009; Zhong et al., 2021).”