Learning Deep Representations of Fine-Grained Visual Descriptions

By Scott E. Reed, Zeynep Akata, Honglak Lee, Bernt Schiele. 2016

923 citations recorded by the source corpus.

State-of-the-art methods for zero-shot visual recognition formulate learning as a joint embedding problem of images and side information. In these formulations the current best complement to visual features are attributes: manually-encoded vectors describing shared characteristics among categories. Despite good performance, attributes have limitations: (1) finer-grained recognition requires commensurately more attributes, and (2) attributes do not provide a natural language interface. We propose to overcome these limitations by training neural language models from scratch, i.e. without pre-training and only consuming words and characters. Our proposed models train end-to-end to align with the fine-grained and category-specific content of images. Natural language provides a flexible and compact way of encoding only the salient visual aspects for distinguishing categories. By training on raw text, our model can do inference on raw text as well, providing humans a familiar mode both for annotation and retrieval. Our model achieves strong performance on zero-shot text-based image retrieval and significantly outperforms the attribute-based state-of-the-art for zero-shot classification on the Caltech-UCSD Birds 200-2011 dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

ALICE: Active Learning with Contrastive Natural Language Explanations (2020, background): “Discussion on CUB Description Dataset The CUB description dataset collects descriptions of visual appearance for each image rather than explanations of why the bird in the image belongs to a certain class (Reed et al., 2016; Hendricks et al., 2016).”
Shaping Visual Representations with Language for Few-Shot Classification (2019, positive): “We use the language descriptions collected by [23], where AMT crowdworkers were asked to describe images of birds in detail, without reference to the species (Figure 2).”
Fine-Grained Image Classification via Combining Vision and Language (2017, methodology): “There are two differences from the work [34]: (1) instead of directly using the GoogleNet, we ﬁrst ﬁne-tune it on the CUB-200-2011, and (2) extract the features of the original image and its object region for each image.”
Generative Adversarial Text to Image Synthesis (2016, methodology): “For both Oxford-102 and CUB we used a hybrid of character-level ConvNet with a recurrent neural network (char-CNN-RNN) as described in (Reed et al., 2016).”
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (2017, methodology): “In addition, a baseline model, ‘AttnGAN1, no attention”, with the text encoder used in [19], is trained on the CUB dataset.”
Prototypical Networks for Few-shot Learning (2017, methodology): “We extracted 1,024-dimensional meta-data vectors for each CUB-200 class using the pretrained Char CNN-RNN model of [25], then trained zero-shot Prototypical Networks using the same procedure described above except we used a 512-dimensional output embedding, as chosen via validation accuracy.”
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks (2016, methodology): “Each image in COCO has 5 descriptions, while 10 descriptions are provided by [25] for every image in CUB and Oxford102 datasets.”
StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks (2017, methodology): “Let ϕ t be the text embedding of the given description, which is generated by a pre-trained encoder [30] in this paper.”
Learning What and Where to Draw (2016, methodology): “In Figure 8 we show several representative examples that we cropped from the supplementary material of [Reed et al., 2016b].”
Generating Visual Explanations (2016, methodology): “We therefore use natural language descriptions [31] which achieved superior performance on zero-shot learning compared to attributes and also shown to be useful for text to image generation [32].”
Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis (2018, methodology): “Conditioned on both the semantic layout M and the text embedding s extracted by [20], the image generator Gimg is jointly trained with the discriminator Dimg.”
Learning to Compare: Relation Network for Few-Shot Learning (2017, methodology): “This DNN is pre-trained on ILSVRC 2012 1K classification without fine-tuning, as in recent deep ZSL works [25, 30, 45].”
Subspace Regularizers for Few-Shot Class Incremental Learning (2021, background): “…from other modalities (especially language) to bootstrap learning of new classes is widely studied (Frome et al., 2013; Radford et al., 2021; Reed et al., 2016; Yan et al., 2021)—particularly in the zero-shot learning and generalized zero-shot learning where side information is the only…”
Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly (2017, background): “[31], [34], [50], [51], [52], [53], [54] which exploit other auxiliary information that reduces this annotation effort.”
Fader Networks: Manipulating Images by Sliding Attributes (2017, methodology): “Since the dataset does not contain other labels than the flower categories, we built a list of color attributes from the flower captions provided by [19].”
TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments (2018, background): “R2R includes 90 unique environments, each containing an average of 119 panoramas, signiﬁcantly smaller than our 29 , 641 panoramas.”
Semantic Image Synthesis via Adversarial Learning (2017, methodology): “We have evaluated our model on Caltech-200 bird dataset [36] and Oxford-102 ﬂower dataset [22], in which ten captions for each image of both datasets are collected by [27].”
Grounding Visual Explanations (2018, background): “Recently, [21] collected 10 sentences for each image with a detailed description of the bird.”