OCNet: Object Context Network for Scene Parsing

By Yuhui Yuan, Jingdong Wang. 2018

648 citations recorded by the source corpus.

In this paper, we address the semantic segmentation task with a new context aggregation scheme named \emph{object context}, which focuses on enhancing the role of object information. Motivated by the fact that the category of each pixel is inherited from the object it belongs to, we define the object context for each pixel as the set of pixels that belong to the same category as the given pixel in the image. We use a binary relation matrix to represent the relationship between all pixels, where the value one indicates the two selected pixels belong to the same category and zero otherwise. We propose to use a dense relation matrix to serve as a surrogate for the binary relation matrix. The dense relation matrix is capable to emphasize the contribution of object information as the relation scores tend to be larger on the object pixels than the other pixels. Considering that the dense relation matrix estimation requires quadratic computation overhead and memory consumption w.r.t. the input size, we propose an efficient interlaced sparse self-attention scheme to model the dense relations between any two of all pixels via the combination of two sparse relation matrices. To capture richer context information, we further combine our interlaced sparse self-attention scheme with the conventional multi-scale context schemes including pyramid pooling~\citep{zhao2017pyramid} and atrous spatial pyramid pooling~\citep{chen2018deeplab}. We empirically show the advantages of our approach with competitive performances on five challenging benchmarks including: Cityscapes, ADE20K, LIP, PASCAL-Context and COCO-Stuff

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

CCNet: Criss-Cross Attention for Semantic Segmentation (2018, methodology): “OCNet [38] and DANet [15] utilized Non-local module [31] to harvest the contextual information.”
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, background): “The selfattention layers can complement backbones [61, 7, 3, 65, 21, 68, 51] or head networks [29, 24] by providing the capability to encode distant dependencies or heterogeneous interactions.”
AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training (2021, methodology): “In order to better aggregate semantic context or feature dependencies, other recent works [41, 10] employ self-attention modules on semantic segmentation networks.”
PLOP: Learning without Forgetting for Continual Semantic Segmentation (2020, background): “Examples of recent works exploiting contextual information include attention mechanisms [69, 76, 22, 30, 68, 61, 73], and fixed-scale aggregation [75, 12, 11, 72].”
Dual Attention Network for Scene Segmentation (2018, methodology): “The concurrent work OCNet [27] adopts self-attention mechanism with ASPP to exploit context dependencies.”
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers (2020, methodology): “For training simplicity, we do not adopt the widely-used tricks such as OHEM [52] loss in model training.”
Deep High-Resolution Representation Learning for Visual Recognition (2019, methodology): “Currently, we have results (mIoU), which are depicted in Tables 3 4 5 6, by combining the HRNet with the object-contextual representation (OCR) scheme [170] 6 , a variant of object context [59], [171].”
Object-Contextual Representations for Semantic Segmentation (2019, methodology): “On the other hand, our OCR approach is also different from the previous relational context schemes [55, 15, 60, 61, 63].”
Segmenter: Transformer for Semantic Segmentation (2021, methodology): “Recent work [22, 23, 57, 58, 61, 66] replace coarse pooling by attention mechanisms on top of the encoder feature maps to better capture long-range dependencies.”
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers (2021, methodology): “After that, researchers focused on improving FCN from different aspects such as: enlarging the receptive ﬁeld [17–19, 5, 2, 4, 20]; reﬁning the contextual information [21–”
ResNeSt: Split-Attention Networks (2020, background): “Cross-channel information has demonstrated success in downstream applications [56, 64, 65], while recent image classiﬁcation networks have focused more on group or depth-wise convolution [27, 28, 54, 60].”
An Empirical Study of Spatial Attention Mechanisms in Deep Networks (2019, background): “Thus, in some recent variants [38, 22, 43, 13], like the non-local block [38] and criss-cross attention module [22], only the query and key content term is kept, with all the other terms removed.”
Hierarchical Multi-Scale Attention for Semantic Segmentation (2020, methodology): “Most recent semantic segmentation work for Cityscapes in particular has utilized the ~20, 000 coarsely labelled images as-is for training state-of-the-art models [12, 29].”
Strip Pooling: Rethinking Spatial Pooling for Scene Parsing (2020, background): “Besides, aggregating non-local context [36, 58, 29, 15, 7, 45, 21, 53, 66, 23, 14] is also effective for scene parsing.”
Image Segmentation Using Deep Learning: A Survey (2020, methodology): “Other applications of attention mechanisms to semantic segmentation include OCNet [92], which employs an object context pooling inspired by self-attention mechanism, ResNeSt: Split-Attention Networks [93], Height-driven Attention Networks [94], Expectation-Maximization Attention (EMANet) [95], Criss”
Unifying Training and Inference for Panoptic Segmentation (2020, methodology): “Our method share similar spirits with random walks [3] and self-attention [28, 12, 24].”
Bidirectional Graph Reasoning Network for Panoptic Segmentation (2020, methodology): “Besides, the scene parsing method with global context information was also studied in [36, 37].”
Prototype-based Incremental Few-Shot Segmentation (2020, background): “State-of-the-art models use a fully convolutional encoder-decoder networks [1, 26], integrate contextual information on pixel-level features in different ways, e.g. through pyramids [5, 6, 7, 23, 59, 60], or attention [13, 51, 52, 54, 55, 57].”
CAP: Context-Aware Pruning for Semantic Segmentation (2021, background): “Lately, for improving accuracy, a few works have begun to exploit the contextual information in semantic segmentation [8, 35, 34, 38], where they mostly attempt to integrate the global context hints with either attention [31] or nonlocal mechanisms [32].”