Context Driven Scene Parsing with Attention to Rare Classes

By Jimei Yang, Brian L. Price, Scott D. Cohen, Ming-Hsuan Yang. 2014

162 citations recorded by the source corpus.

This paper presents a scalable scene parsing algorithm based on image retrieval and superpixel matching. We focus on rare object classes, which play an important role in achieving richer semantic understanding of visual scenes, compared to common background classes. Towards this end, we make two novel contributions: rare class expansion and semantic context description. First, considering the long-tailed nature of the label distribution, we expand the retrieval set by rare class exemplars and thus achieve more balanced superpixel classification results. Second, we incorporate both global and local semantic context information through a feedback based mechanism to refine image retrieval and superpixel matching. Results on the SIFTflow and LMSun datasets show the superior performance of our algorithm, especially on the rare classes, without sacrificing overall labeling accuracy.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Interactive Object Selection (2016, methodology): “Many algorithms have been proposed such as superpixel-based [28, 24, 25] and clusteringbased methods [23, 15, 14].”
Scene Parsing through ADE20K Dataset (2017, background): “For example, MRF frameworks are proposed to parse images in different levels [30] or segment rare object classes [33]; detection is combined with segmentation to improve the performance [31]; stuff classes are leveraged to localize objects [13].”
Semantic Image Segmentation via Deep Parsing Network (2015, background): “Global or local semantic contexts between labels were also investigated by [38].”
Strip Pooling: Rethinking Spatial Pooling for Scene Parsing (2020, background): “One way to improve the capability of modeling the long-range dependencies in CNNs is to adopt self-attention or non-local modules [51, 23, 7, 45, 21, 53, 66, 62, 61, 28].”
Not All Pixels Are Equal: Difficulty-Aware Semantic Segmentation via Deep Layer Cascade (2017, background): “While early efforts focused on structural models with handcrafted features [15, 16, 34, 38], recent studies employ deep convolutional neural network (CNN) to learning strong representation, which improves segmentation accuracy signiﬁcantly [3, 22, 23, 25, 40].”
Factors in Finetuning Deep Model for Object Detection with Long-Tail Distribution (2016, background): “[44] expand the samples of rare classes and achieve more balanced superpixel classification results.”
IDD: A Dataset for Exploring Problems of Autonomous Navigation in Unconstrained Environments (2018, background): “Besides the presence of rare categories [24], even common categories have diverse attributes or appearance variations.”
Learning to segment under various forms of weak supervision (2015, methodology): “Semantic segmentation is one of the most fundamental challenges in computer vision, and conventional fully supervised algorithms have demonstrated promising results [21, 10, 8, 33, 35, 36, 44, 19, 20, 45, 31].”
Episodic CAMN: Contextual Attention-Based Memory Networks with Iterative Feedback for Scene Labeling (2017, methodology): “We found that applying class balancing helps similar to [51, 67].”
Integrating parametric and non-parametric models for scene labeling (2015, background): “Recently, non-parametric label transfer methods [3, 16, 24, 25, 29, 34] have gained popularity due to its outstanding performance and scalability to increasing data.”
Range Loss for Deep Face Recognition with Long-Tailed Training Data (2016, background): “Long tailed distribution of the data has been studied in scene parsing [33], and zero-shot learning [18].”
Deep Long-Tailed Learning: A Survey (2021, background): “Non-deep long-tailed learning: There are a lot of non-deep learning approaches for long-tailed problems [65], [66], [67].”
REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets (2020, background): “Object counts in the real world tend to naturally follow a long-tail distribution (Ouyang et al., 2016; Salakhut-dinov et al., 2011; J. Yang et al., 2014).”
A Spatial-Temporal Attention-Based Method and a New Dataset for Remote Sensing Image Change Detection (2020, background): “Originating from the human visual system [48], attention mechanism models the dependencies between input and output sequences and has been applied in various tasks, such as neural machine translation [49], image captioning [50] and scene parsing [51].”
Deep hierarchical parsing for semantic segmentation (2015, background): “These approaches are instances of sophisticated template matching to retrieve images that are visually similar to the query, from a database of labeled images.”
Region-Based Semantic Segmentation with End-to-End Training (2016, methodology): “Most older works use hand-crafted region-based features [1, 3, 5, 8, 10, 13–16] often based on [3, 13].”
Sensor fusion for semantic segmentation of urban scenes (2015, methodology): “For images, Tighe et al. [6] [7], Singh et al. [8], and Yang et al. [9] use an image retrieval system followed by non-parametric classification to produce superpixel classifications, with special considerations taken for objects [7], semantic context [8], and rare classes [9].”
Dense Recurrent Neural Networks for Scene Labeling (2018, background): “Early efforts mainly focus on the probabilistic graphical model with hand-crafted features [30, 44, 49, 17].”