Open Vocabulary Scene Parsing

By Hang Zhao, Xavier Puig, Bolei Zhou, Sanja Fidler, Antonio Torralba. 2017

144 citations recorded by the source corpus.

Recognizing arbitrary objects in the wild has been a challenging problem due to the limitations of existing classification models and datasets. In this paper, we propose a new task that aims at parsing scenes with a large and open vocabulary, and several evaluation metrics are explored for this problem. Our approach is a joint image pixel and word concept embeddings framework, where word concepts are connected by semantic relations. We validate the open vocabulary prediction ability of our framework on ADE20K dataset which covers a wide variety of scenes and objects. We further explore the trained joint embedding space to show its interpretability.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Open-vocabulary Object Detection via Vision and Language Knowledge Distillation (2021, background): “…direction, researchers have also explored class hierarchy, class similarity, and object parts as discriminative features to aid the knowledge transfer (Rohrbach et al., 2011; Akata et al., 2016; Zhao et al., 2017; Elhoseiny et al., 2017; Ji et al., 2018; Cacheux et al., 2019; Xie et al., 2020).”
ConceptFusion: Open-set Multimodal 3D Mapping (2023, background): “This has led to a series of approaches that extract pixel-aligned features [24, 25, 18, 26, 23]1 that address poor localization quality by training or fine-tuning a pixel-aligned model over a labeled dataset.”
Open-Vocabulary Image Segmentation (2021, methodology): “[52] propose to use WordNet hierarchy to aid knowledge transfer.”
Procedure Planning in Instructional Videos (2019, background): “ understanding [3] and task planning [6], simultaneously addressing both in real-world instructional videos remains unsolved. The key challenge of the task described in Figure1 is the open-vocabulary [5, 7, 25] state and action spaces in real-world instructional videos. For example, the current obse”
Zero-Shot Semantic Segmentation via Variational Mapping (2019, background): “Zhao et al. [36] propose open vocabulary scene parsing.”
Semantic Projection Network for Zero- and Few-Label Semantic Segmentation (2019, background): “The term zeroshot semantic segmentation appears in prior works [18, 58].”
What the DAAM: Interpreting Stable Diffusion Using Cross Attention (2022, background): “One promising line of future work is to extend DAAM to work in an unsupervised manner on open-vocabulary semantic segmentation on real images [24].”
Dynamic-Structured Semantic Propagation Network (2018, methodology): “We further compare our DSSPN with prior works that also tried the hierarchical classiﬁcation [31, 10, 39] based on pretrained VGG net, as reported in [39].”
Uncertainty-Aware Learning for Zero-Shot Semantic Segmentation (2020, methodology): “To achieve effective zero-shot semantic segmentation (ZSS), existing efforts have been made [2, 28, 31, 44, 55, 62] by treating each pixel as an independent classiﬁcation problem.”
Large-Scale Visual Relationship Understanding (2018, background): “Very recent approaches like [40, 27] target open-vocabulary for scene parsing and visual relationship detection, respectively.”
Hierarchical Novelty Detection for Visual Object Recognition (2018, background): “Incorporating the hierarchical taxonomy for object classification has been investigated in the literature, either to improve classification performance [6, 34], or to extend the classification tasks to obtain more informative results [8, 36].”
Decoupling Zero-Shot Semantic Segmentation (2021, background): “Zero-Shot Semantic Segmentation is a relatively new research topic [7, 25, 42, 54, 61], which aims to segment the classes that have not been seen during training.”
Side Adapter Network for Open-Vocabulary Semantic Segmentation (2023, background): “Open-vocabulary Semantic Segmentation Earlier work [3, 31, 40] for open-vocabulary semantic segmentation focus on learning a joint embedding space between image pixels and class name/description.”
Extract Free Dense Labels from CLIP (2021, background): “[54] project image pixel features and word concepts into a joint space.”
Symbolic Graph Reasoning Meets Convolutions (2018, methodology): “Some works regularized the output of networks by resorting to complex graphical inference [9], hierarchical loss [38] or word embedding priors [49] on final prediction scores.”
Scaling Open-Vocabulary Image Segmentation with Image-Level Labels (2021, background): “[59] is the pioneering work that learns a joint pixel and word concept embedding space; however, its vocabulary and knowledge is limited to WordNet and can not take arbitrary texts as input.”
Context-aware Feature Generation For Zero-shot Semantic Segmentation (2020, methodology): “Zero-shot Semantic Segmentation: The term zero-shot semantic segmentation appeared in prior works [3, 17, 43, 49], in which only SPNet [43] and ZS3Net [3] focused on multi-category semantic segmentation.”
WildFish: A Large Benchmark for Fish Recognition in the Wild (2018, background): “To alleviate this difficulty, attention has been proposed recently by two-level domain net [44], reinforcement learning [27], diversified design [49], multi-attention for part learning [50], and recurrent attention [14].”