Semantic Object Parsing with Graph LSTM

By Xiaodan Liang, Xiaohui Shen, Jiashi Feng, Liang Lin, Shuicheng Yan. 2016

369 citations recorded by the source corpus.

By taking the semantic object parsing task as an exemplar application scenario, we propose the Graph Long Short-Term Memory (Graph LSTM) network, which is the generalization of LSTM from sequential data or multi-dimensional data to general graph-structured data. Particularly, instead of evenly and fixedly dividing an image to pixels or patches in existing multi-dimensional LSTM structures (e.g., Row, Grid and Diagonal LSTMs), we take each arbitrary-shaped superpixel as a semantically consistent node, and adaptively construct an undirected graph for each image, where the spatial relations of the superpixels are naturally used as edges. Constructed on such an adaptive graph topology, the Graph LSTM is more naturally aligned with the visual patterns in the image (e.g., object boundaries or appearance similarities) and provides a more economical information propagation route. Furthermore, for each optimization step over Graph LSTM, we propose to use a confidence-driven scheme to update the hidden and memory states of nodes progressively till all nodes are updated. In addition, for each node, the forgets gates are adaptively learned to capture different degrees of semantic correlation with neighboring nodes. Comprehensive evaluations on four diverse semantic object parsing datasets well demonstrate the significant superiority of our Graph LSTM over other state-of-the-art solutions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (2016, methodology): “Employing the dense CRF to post process our final output substantially outperforms the concurrent work [97] by 4.”
Sentence-State LSTM for Text Representation (2018, methodology): “, 2016) and image object detection (Liang et al., 2016).”
Dual Attention Network for Scene Segmentation (2018, methodology): “The work (Liang et al. 2016) proposes a Graph LSTM to deal with general graph-structured data in semantic object parsing tasks.”
RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation (2016, methodology): “Qualitative examples of our object parsing on this dataset are shown in Fig.4.”
Scene Graph Generation by Iterative Message Passing (2017, methodology): “Our work is closely related to CRFasRNN [43] and Graph-LSTM [23] in that we also formulate the graph inference problem using an RNN-based model.”
Image Segmentation Using Deep Learning: A Survey (2020, neutral): “[85] proposed a semantic segmentation model based on a graph-LSTM network (Fig.”
Graph Neural Networks: A Review of Methods and Applications (2018, background): “[49] proposed a Graph LSTM network to address the semantic object parsing task.”
Dense Recurrent Neural Networks for Scene Labeling (2018, background): “[27] propose a graph based LSTM to model the dependencies among superpixels in images.”
Instance-level Human Parsing via Part Grouping Network (2018, neutral): “59 56.11 Attention [5] 81.47 59.06 44.15 42.50 38.28 35.62 93.65 56.39 LG-LSTM [20] 82.72 60.99 45.40 47.76 42.33 37.96 88.63 57.97 LIP [11] 83.26 62.40 47.80 45.58 42.32 39.48 94.68 59.36 Graph LSTM [19] 82.69 62.68 46.88 47.71 45.66 40.93 94.59 60.16 Structure-evolving LSTM [17] 82.89 67.15 51.42 ”
GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs (2018, methodology): “Models like the adaptive forget gate strategy in Graph LSTM (Liang et al., 2016) and MoNet (Monti et al., 2017) employed pairwise sum ag-gregators with a single head or multiple heads.”
Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling (2020, methodology): “2020), and then perform inference with graph neural models (Liang et al. 2016; Guo, Zhang, and Lu 2019).”
GTR-LSTM: A Triple Encoder for Sentence Generation from RDF Data (2018, methodology): “Meanwhile, Graph LSTM only allows a predeﬁned set of relationships between entities, while the proposed model allows any relationships by treating them as part of the input for the hidden state computation.”
High-Order Graph Convolutional Recurrent Neural Network: A Deep Learning Framework for Network-Scale Traffic Learning and Forecasting (2018, background): “(4) LSTM: Long Short-Term Memory recurrent neural network [7]; (5) GLSTM: Graph LSTM [11]; (6) HGC-LSTM: proposed by this work without the RTBL training strategy.”
Residual Gated Graph ConvNets (2017, methodology): “The proposed Graph LSTM model differs from Liang et al. (2016); Peng et al. (2017) mostly because cell C G-LSTM in these previous models is not iterated over multiple times T , which reduces the performance of Graph LSTM (see numerical experiments on Figure 4).”
Dynamic-Structured Semantic Propagation Network (2018, methodology): “Instead of using single instance training in most of treestructured [34] and graph-structured [21, 19] networks, our DSSPN uses dynamic graph batching strategy to make good use of efficient data-parallel algorithms and hardware, inspired by very recent attempts [25, 3, 26].”
Large-Scale Point Cloud Semantic Segmentation with Superpoint Graphs (2017, background): “[32, 31] traverses such graphs in a sequential node order based on unary confidences to improve the final labels.”
DyNet: The Dynamic Neural Network Toolkit (2017, background): “Variably structured inputs: A more complicated case is when each input has not only a diﬀerent size, but also a diﬀerent structure, such as the case of tree-structured networks [55, 61] or graph-structured networks [41], which have been used in a variety of tasks.”