Joint Object and Part Segmentation Using Deep Learned Potentials

By Peng Wang, Xiaohui Shen, Zhe L. Lin, Scott D. Cohen, Brian L. Price, Alan L. Yuille. 2015

128 citations recorded by the source corpus.

Segmenting semantic objects from images and parsing them into their respective semantic parts are fundamental steps towards detailed object understanding in computer vision. In this paper, we propose a joint solution that tackles semantic object and part segmentation simultaneously, in which higher object-level context is provided to guide part segmentation, and more detailed part-level localization is utilized to refine object segmentation. Specifically, we first introduce the concept of semantic compositional parts (SCP) in which similar semantic parts are grouped and shared among different objects. A two-stream fully convolutional network (FCN) is then trained to provide the SCP and object potentials at each pixel. At the same time, a compact set of segments can also be obtained from the SCP predictions of the network. Given the potentials and the generated segments, in order to explore long-range context, we finally construct an efficient fully connected conditional random field (FCRF) to jointly predict the final object and part labels. Extensive evaluation on three different datasets shows that our approach can mutually enhance the performance of object and part segmentation, and outperforms the current state-of-the-art on both tasks.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (2016, methodology): “We further perform experiments on semantic part segmentation [98], [99], using the extra PASCAL VOC 2010 annotations by [36].”
Attention to Scale: Scale-Aware Semantic Image Segmentation (2015, background): “Few works [51, 52] have worked on the animal part segmentation for the dataset.”
DeepVoting: An Explainable Framework for Semantic Part Detection under Partial Occlusion (2017, methodology): “On the other hand, semantic part detection, despite its importance, has been much less studied (Wang et al. 2015a).”
Semantic Object Parsing with Graph LSTM (2016, methodology): “To capture the rich structure information based on the advanced CNN architecture, one common way is the combination of CNNs and CRFs [11][6][37][12], where the CNN outputs are treated as unary potentials while CRF further incorporates pairwise or higher order factors.”
Look into Person: Self-Supervised Structure-Sensitive Learning and a New Benchmark for Human Parsing (2017, background): “Besides human parsing, there has also been increasing research interest on the part segmentation of other objects such as animals or cars [26, 27, 22].”
Semantic Object Parsing with Local-Global Long Short-Term Memory (2015, methodology): “Previous works often resort to some post-processing techniques to separately address these complex contextual dependencies, such as, super-pixel smoothing [15], mean field approximation [22] and conditional random field [3][38][33].”
Zoom Better to See Clearer: Human Part Segmentation with Auto Zoom Net (2015, methodology): “In addition, human parsing can be widely applicable to other computer vision tasks, e.g., segmentation (Eslami & Williams, 2012; Wang et al., 2015a), pose estimation (Dong et al., 2014), fine-grained recognition (Zhang et al., 2014), and industry applications such as robotics and image descriptions…”
The ApolloScape Open Dataset for Autonomous Driving and Its Application (2018, background): “we consider high requirements for object boundaries rather than class accuracy, for each bounding box with high confidence ( 0:9), we enlarge the bounding box and crop out the object region with context similar to [78].”
Zoom Better to See Clearer: Human and Object Parsing with Hierarchical Auto-Zoom Net (2015, methodology): “In computer vision, object parsing plays a key role for real understanding of objects in images and helps for many visual tasks, e.g., segmentation [9,30], pose estimation [8], and ﬁne-grained recognition [35].”
Look into Person: Joint Body Parsing & Pose Estimation Network and a New Benchmark (2018, background): “, conditional random fields (CRFs)), similar to the general object segmentation approaches [12], [23], [24].”
Affinity CNN: Learning Pixel-Centric Pairwise Relations for Figure/Ground Embedding (2015, background): “…a generalized eigenproblem: Wz = λDz (4) with D and W defined in terms of C and Θ by: D = Diag(C1n) (5) W = C • eiΘ (6) where n is the number of pixels, 1n is a column vector of ones, Diag(·) is a matrix with its vector argument on the main diagonal, and • denotes the matrix Hadamard product.”