Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models
By Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, Stephen Gould. IEEE International Conference on Computer Vision, 2021
373 citations recorded by the source corpus.
We extend the task of composed image retrieval, where an input query consists of an image and short textual description of how to modify the image. Existing methods have only been applied to non-complex images within narrow domains, such as fashion products, thereby limiting the scope of study on in-depth visual reasoning in rich image and language contexts. To address this issue, we collect the Compose Image Retrieval on Real-life images (CIRR) dataset, which consists of over 36,000 pairs of crowd-sourced, open-domain images with human-generated modifying text. To extend current methods to the open-domain, we propose CIRPLANT, a transformer based model that leverages rich pre-trained vision-and-language (V&L) knowledge for modifying visual features conditioned on natural language. Retrieval is then done by nearest neighbor lookup on the modified features. We demonstrate that with a relatively simple architecture, CIRPLANT outperforms existing methods on open-domain images, while matching state-of-the-art accuracy on the existing narrow datasets, such as fashion. Together with the release of CIRR, we believe this work will inspire further research on composed image retrieval. Our dataset, code and pre-trained models are available at https://cuberick-orion.github.io/CIRR/.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/c2d5426ee019d3d894b9d4416dc866b65fe64312
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
- Ziqi Zhou
- Lingqiao Liu
- Yinjie Lei
- Yifan Liu
- Chunhua Shen
- Andrea Vedaldi
- Anton van den Hengel
- Subhransu Maji
Citations in the corpus
- Multimodal Learning With Transformers: A Survey (2022, neutral): “In addition to common caption and QA datasets, more applications and scenarios have been studied, e.g., CIRR [149] (real-life images), Product1M [137], Bed and Breakfast (BnB) [150] (vision-and-language navigation), M3A [151] (financial dataset), X-World [152] (autonomous drive).”
- ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation (2022, background): “Pre-trained Vision Language Models [19, 22, 36, 40] which connects the image representation with text embedding have achieved significant performance on various downstream tasks, like image retrieval [28], dense prediction [37], visual referring expression [43], visual question answering [21] and so”
- MM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval (2026, unknown): “Early benchmarks like WebQA [5] and CIRR [12] focused on cross-modal semantic matching or multi-hop QA, often relying on simple visual similarity.”
- Towards Text-Image Interleaved Retrieval (2025, unknown): “The scope has been extended to more complex scenarios, such as composed image retrieval (Liu et al., 2021), scientific contents (Wu et al., 2024), and visual documents (Ma et al., 2024; Faysse et al., 2025).”