Harmony potentials for joint classification and segmentation

By Josep M. Gonfaus, Xavier Boix, Joost van de Weijer, Andrew D. Bagdanov, Joan Serrat Gual, Jordi Gonzàlez. 2010

185 citations recorded by the source corpus.

Hierarchical conditional random fields have been successfully applied to object segmentation. One reason is their ability to incorporate contextual information at different scales. However, these models do not allow multiple labels to be assigned to a single node. At higher scales in the image, this yields an oversimplified model, since multiple classes can be reasonable expected to appear within one region. This simplified model especially limits the impact that observations at larger scales may have on the CRF model. Neglecting the information at larger scales is undesirable since class-label estimates based on these scales are more reliable than at smaller, noisier scales. To address this problem, we propose a new potential, called harmony potential, which can encode any possible combination of class labels. We propose an effective sampling strategy that renders tractable the underlying optimization problem. Results show that our approach obtains state-of-the-art results on two challenging datasets: Pascal VOC 2009 and MSRC-21.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (2016, methodology): “of segments [29], [30], [31], [32], [33], we use the fully connected pairwise CRF proposed by [22] for its efficient computation, and ability to capture fine edge details while also catering for long range dependencies.”
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs (2014, methodology): “…with the spatial localization issues outlined in the beginning of the introduction, Long et al. (2014) upsample and concatenate the scores from inter-mediate feature maps, while Eigen & Fergus (2014) reﬁne the prediction result from coarse to ﬁne by propagating the coarse results to another DCNN.”
Higher Order Conditional Random Fields in Deep Neural Networks (2015, methodology): “Numerous improvements to this model were subsequently proposed including: densely connected pairwise potentials facilitating interactions between all pairs of image pixels [24], formulating higher order potentials defined over cliques larger than two nodes [21,27] in order to capture more context, m”
What, Where and How Many? Combining Object Detectors and CRFs (2010, methodology): “Both the BONN [34] and CVC [35] methods can be directly placed in our work, and should lead to an increase in performance.”
Relating Things and Stuff by High-Order Potential Modeling (2012, methodology): “We model different types of pair-wise stuff relationships using a codebook representation similar to [23].”
Top-down visual saliency via joint CRF and dictionary learning (2012, background): “CRFs have been demonstrated as a flexible framework of incorporating different kinds of features for visual recognition [25, 24, 5, 7, 1].”
CRF Learning with CNN Features for Image Segmentation (2015, unknown): “[14] 77 75 Lucchi et al.”
Describing the scene as a whole: Joint object detection, scene classification and semantic segmentation (2012, background): “In [2], a global image node was introduced.”
Proposal generation for object detection using cascaded ranking SVMs (2011, methodology): “Ranking SVMs have been used recently in object detection [2] and segmentation [3, 11, 15].”
Exploring Person Context and Local Scene Context for Object Detection (2015, background): “CRFbased approaches have tried to capture both the local consistency between adjacent pixels or detections and the consistency between detections and the entire scene [12, 33].”
Structured class-labels in random forests for semantic image labelling (2011, methodology): “For the task of semantic image labelling, state-of-the-art approaches [16, 17, 13, 24] are typically using complementary features at different levels within random field models [18].”
Multi-objective convolutional learning for face labeling (2015, background): “Conditional random fields (CRFs) are another important class of image labeling models [14, 5, 6, 2] that carry out structured prediction by considering label dependencies and allow flexible use of pre-trained image features.”
Learning a Dictionary of Shape Epitomes with Applications to Image Labeling (2013, background): “The patches are overlapping, thus allowing neighbors to directly communicate with each other and find configurations which are consistent in their area of overlap (Model-1).”
Real-Time Online Action Detection Forests Using Spatio-Temporal Contexts (2016, methodology): “3) for each split function Ψ enables the resulting decision tree to take account high-order information similarly to CRFs: It has been demonstrated that CRF’s high-order potentials can help to identify the underlying structure in the output space [13, 20, 23] and thereby offer higher classification ”