Scaling Open-Vocabulary Image Segmentation with Image-Level Labels
By Golnaz Ghiasi, Xiuye Gu, Yin Cui, Tsung-Yi Lin. 2021
614 citations recorded by the source corpus.
We design an open-vocabulary image segmentation model to organize an image into meaningful regions indicated by arbitrary texts. Recent works (CLIP and ALIGN), despite attaining impressive open-vocabulary classification accuracy with image-level caption labels, are unable to segment visual concepts with pixels. We argue that these models miss an important step of visual grouping, which organizes pixels into groups before learning visual-semantic alignments. We propose OpenSeg to address the above issue while still making use of scalable image-level supervision of captions. First, it learns to propose segmentation masks for possible organizations. Then it learns visual-semantic alignments by aligning each word in a caption to one or a few predicted masks. We find the mask representations are the key to support learning image segmentation from captions, making it possible to scale up the dataset and vocabulary sizes. OpenSeg significantly outperforms the recent open-vocabulary method of LSeg by +19.9 mIoU on PASCAL dataset, thanks to its scalability.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Ziwei Liu
- Chen Change Loy
- Kai Chen
- Josh Tenenbaum
- Antonio Torralba
- Shuang Li
- Jiangmiao Pang
- Wenwei Zhang
In Computer Vision and Pattern Recognition
- Chen Change Loy
- Luc Van Gool
- Xiaogang Wang
- Ming-Hsuan Yang
- Dacheng Tao
- Jitendra Malik
- Andrea Vedaldi
- Trevor Darrell
Citations in the corpus
- ConceptFusion: Open-set Multimodal 3D Mapping (2023, methodology): “In the first two scenarios, OpenSeg [17] is distracted by the presence of several confounding attributes (kitchen counter, window, television).”
- Transformer-Based Visual Segmentation: A Survey (2023, methodology): “OpenSeg[281]learnstogeneratesegmentationmasks for possible candidates using a DETR-like transformer.”
- OpenMask3D: Open-Vocabulary 3D Instance Segmentation (2023, methodology): “9 Open-vocabulary OpenScene [49] (2D Fusion) - OpenSeg [18] 11.”
- Vision-Language Models for Vision Tasks: A Survey (2023, background): “For example, [35], [180], [181] achieve open-vocabulary semantic segmentation by first class-agnostic segmentation by grouping pixels into multiple segments and then segment recognition with CLIP.”