ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension
By Sanjay Subramanian, William Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, Anna Rohrbach. 2022
187 citations recorded by the source corpus.
Training a referring expression comprehension (ReC) model for a new visual domain requires collecting referring expressions, and potentially corresponding bounding boxes, for images in the domain. While large-scale pre-trained models are useful for image classification across domains, it remains unclear if they can be applied in a zero-shot manner to more complex tasks like ReC. We present ReCLIP, a simple but strong zero-shot baseline that repurposes CLIP, a state-of-the-art large-scale model, for ReC. Motivated by the close connection between ReC and CLIP’s contrastive pre-training objective, the first component of ReCLIP is a region-scoring method that isolates object proposals via cropping and blurring, and passes them to CLIP. However, through controlled experiments on a synthetic dataset, we find that CLIP is largely incapable of performing spatial reasoning off-the-shelf. We reduce the gap between zero-shot baselines from prior work and supervised models by as much as 29% on RefCOCOg, and on RefGTA (video game imagery), ReCLIP’s relative improvement over supervised ReC models trained on real images is 8%.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Cordelia Schmid
- Trevor Darrell
- Antonio Torralba
- Andrew Zisserman
- Mohit Bansal
- Sergey Levine
- Josh Tenenbaum
- Li Fei-Fei
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Trevor Darrell
- Andrew Zisserman
- Cordelia Schmid
- Jitendra Malik
- Xiaogang Wang
- Antonio Torralba
- Dacheng Tao
In Representation & Self-Supervised Learning
- Cordelia Schmid
- Antonio Torralba
- Trevor Darrell
- Josh Tenenbaum
- Arsha Nagrani
- Kristen Grauman
- Andrea Vedaldi
- Kevin Yang
Citations in the corpus
- Visual Spatial Reasoning (2022, background): “We only report direct prompting results without finetuning since CLIP finetuning is expensive.”
- ConceptFusion: Open-set Multimodal 3D Mapping (2023, background): “As investigated in [82, 83, 73], CLIP does not inherently capture spatial relationships or compositions.”
- Modular Visual Question Answering via Code Generation (2023, methodology): “Subramanian et al. (2022) used a syntactic parser and hard-coded rules rather than an LM to aggregate outputs from CLIP (Radford et al., 2021) for zero-shot referring expression comprehension; their finding that CLIP is not useful for spatial keywords motivates our code generation approach to…”