Semantic Object Parsing with Local-Global Long Short-Term Memory

By Xiaodan Liang, Xiaohui Shen, Donglai Xiang, Jiashi Feng, Liang Lin, Shuicheng Yan. 2015

188 citations recorded by the source corpus.

Semantic object parsing is a fundamental task for understanding objects in detail in computer vision community, where incorporating multi-level contextual information is critical for achieving such fine-grained pixel-level recognition. Prior methods often leverage the contextual information through post-processing predicted confidence maps. In this work, we propose a novel deep Local-Global Long Short-Term Memory (LG-LSTM) architecture to seamlessly incorporate short-distance and long-distance spatial dependencies into the feature learning over all pixel positions. In each LG-LSTM layer, local guidance from neighboring positions and global guidance from the whole image are imposed on each position to better exploit complex local and global contextual information. Individual LSTMs for distinct spatial dimensions are also utilized to intrinsically capture various spatial layouts of semantic parts in the images, yielding distinct hidden and memory cells of each position for each dimension. In our parsing approach, several LG-LSTM layers are stacked and appended to the intermediate convolutional layers to directly enhance visual features, allowing network parameters to be learned in an end-to-end way. The long chains of sequential computation by stacked LG-LSTM layers also enable each pixel to sense a much larger region for inference benefiting from the memorization of previous dependencies in all positions along all dimensions. Comprehensive evaluations on three public datasets well demonstrate the significant superiority of our LG-LSTM over other state-of-the-art methods.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Rethinking Atrous Convolution for Semantic Image Segmentation (2017, methodology): “There are other methods based on LSTM [31] to aggregate global context [46, 6, 79].”
RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation (2016, methodology): “Qualitative examples of our object parsing on this dataset are shown in Fig.4.”
Reversible Recursive Instance-Level Object Segmentation (2015, background): “Recently, beyond the traditional object detection [7][29][5][15][4][30] and semantic segmentation tasks [1][23][2][37][19][18][21][20][14][16], instancelevel object segmentation has attracted much attention [10][11][22][31][36][17].”
Image-Based Localization Using LSTMs for Structured Feature Correlation (2016, methodology): “[53] uses spatial LSTM for person re-identification, parsing the detection bounding box horizontally and vertically in order to capture spatial dependencies between body parts, [6] employs the same idea for semantic segmentation, and [32] for semantic object parsing.”
Gated Feedback Refinement Network for Dense Image Labeling (2017, unknown): “31 LG - LSTM [19] 89.”
Look into Person: Self-Supervised Structure-Sensitive Learning and a New Benchmark for Human Parsing (2017, background): “Recently, Convolutional Neural Networks (CNNs) have achieved exciting success in human parsing [13, 17, 15].”
Instance-level Human Parsing via Part Grouping Network (2018, unknown): “-Person-Part [6]. Method head torso u-arms l-arms u-legs l-legs Bkg Avg HAZN [39] 80.79 59.11 43.05 42.76 38.99 34.46 93.59 56.11 Attention [5] 81.47 59.06 44.15 42.50 38.28 35.62 93.65 56.39 LG-LSTM [20] 82.72 60.99 45.40 47.76 42.33 37.96 88.63 57.97 LIP [11] 83.26 62.40 47.80 45.58 42.32 39.48 94”
Look into Person: Joint Body Parsing & Pose Estimation Network and a New Benchmark (2018, background): “Recently, convolutional neural networks (CNNs) have achieved exciting success in human parsing [6], [7], [8] and pose estimation [9], [10].”
3D Graph Neural Networks for RGBD Semantic Segmentation (2017, methodology): “[22, 23] tackled semantic segmentation as sequence prediction and used LSTM to capture local and global dependencies.”
Semantic Object Parsing with Graph LSTM (2016, methodology): “For fair comparing with [17][36] on two human parsing datasets, the basic “Co-CNN” structure proposed in [36] is utilized due to its leading accuracy.”
Recent Advances in Recurrent Neural Networks (2017, methodology): “This technique directly enhances visual features and allows an end-to-end learning of network parameters [66].”
Dense and Low-Rank Gaussian CRFs Using Deep Embeddings (2017, methodology): “The state of the art approaches on human part segmentation also use fully convolutional networks, sometimes additionally exploiting Long Short Term Memory Units [24, 25].”
Spatial As Deep: Spatial CNN for Traffic Scene Understanding (2017, methodology): “d Bell et al. (2016) used recurrent neural networks to pass information along each row or column, thus in one RNN layer each pixel position could only receive information from the same row or column. Liang et al. (2016a; 2016b) proposed variants of LSTM to exploit contextual information in semantic ”
Self-Correction for Human Parsing (2019, unknown): “54 LG-LSTM [20] 82.”