Composing Text and Image for Image Retrieval - an Empirical Odyssey

By Nam S. Vo, Lu Jiang, Chen Sun, Kevin P. Murphy, Li-Jia Li, Li Fei-Fei, James Hays. Computer Vision and Pattern Recognition, 2018

489 citations recorded by the source corpus.

In this paper, we study the task of image retrieval, where the input query is specified in the form of an image plus some text that describes desired modifications to the input image. For example, we may present an image of the Eiffel tower, and ask the system to find images which are visually similar, but are modified in small ways, such as being taken at nighttime instead of during the day. o tackle this task, we embed the query (reference image plus modification text) and the target (images). The encoding function of the image text query learns a representation, such that the similarity with the target image representation is high iff it is a ``positive match''. We propose a new way to combine image and text through residual connection, that is designed for this retrieval task. We show this outperforms existing approaches on 3 different datasets, namely Fashion-200k, MIT-States and a new synthetic dataset we create based on CLEVR. We also show that our approach can be used to perform image classification with compositionally novel labels, and we outperform previous methods on MIT-States on this task.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/fd5129e8ebfaa5dcce3d4ce2839b90c6cd3ca39d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback (2021, methodology): “Recently, several techniques have been proposed for image or video retrieval based on natural language queries [36, 4, 60, 65, 55].”
RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval (2020, background): “able with the conditional generative model. Image Retrieval. Image retrieval has been a classical vision problem with numerous applications such as product search [24,6,1], multimodal image retrieval [36,3], image geolocalization [7], event detection [13], among others. Solutions based on deep metri”
A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2021, background): “, text-to-image retrieval [100], sketch-to-image retrieval [101], cross-view image retrieval [102], composing text and image-to-image [103], etc.”
Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision (2020, background): “Most interestingly, we show that they can be also be generated from existing annotations [18,25,36,61,62,70].”
Text as Neural Operator:Image Manipulation by Text Instruction (2020, methodology): “cross-modal image retrieval, a fundamental problem in information retrieval for its crucial role in a variety of industrial applications such as product search [23, 54, 11]. In this retrieval setting [47], users search an image database using an input query that is formed of an image plus some text ”
Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models (2021, methodology): “Specifically, our design maintains the same objective of “language-conditioned image feature modification” as previous work [5, 8, 40], while manages to utilize the pre-trained V&L knowledge in large-scale models.”