Crisp Boundary Detection Using Pointwise Mutual Information

By Phillip Isola, Daniel Zoran, Dilip Krishnan, Edward H. Adelson. 2014

203 citations recorded by the source corpus.

Detecting boundaries between semantically meaningful objects in visual scenes is an important component of many vision algorithms. In this paper, we propose a novel method for detecting such boundaries based on a simple underlying principle: pixels belonging to the same object exhibit higher statistical dependencies than pixels belonging to different objects. We show how to derive an affinity measure based on this principle using pointwise mutual information, and we show that this measure is indeed a good predictor of whether or not two pixels reside on the same object. Using this affinity with spectral clustering, we can find object boundaries in the image – achieving state-of-the-art results on the BSDS500 dataset. Our method produces pixel-level accurate boundaries while requiring minimal feature engineering.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

InstanceCut: From Edges to Instances with MultiCut (2016, methodology): “More recent methods are based on spectral clustering [47, 1, 3, 26].”
EDTER: Edge Detection with Transformer (2022, methodology): “We compare our model with traditional detectors including Canny [6], Felz-Hutt [18], gPb-owt-ucm [2], SCG [64], Sketch Tokens [34], PMI [25], SE [15], OEF [21] and MES [50], and deep-learning-based detectors including DeepEdge [3], CSCNN [24], DeepCon-tour [48], HFL [4], HED [65], Deep Boundary [29]”
Learning visual groups from co-occurrences in space and time (2015, background): “Isola et al. (2014) showed that statistical dependences between adjacent pixel colors, measured by pointwise mutual information (PMI) can be very effective at localizing object boundaries.”
On the Binding Problem in Artificial Neural Networks (2020, methodology): “One approach is to learn a similarity function between pairs of pixels, e.g. based on their point-wise mutual information using kernel-density estimation (Isola et al., 2014) or based on self-supervised prediction using a neural network (Isola et al., 2015).”
DeepEdge: A multi-scale bifurcated deep network for top-down contour detection (2014, background): “A detailed survey of edge detection methods can be found in [4, 30].”
N4-Fields: Neural Network Nearest Neighbor Fields for Image Transforms (2014, methodology): “In general, N 4 -ﬁelds perform similarly to the best previously published methods [20, 10, 26].”
High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and Its Applications to High-Level Vision (2015, methodology): “The most successful approaches in this genre are the MCG detector [2], gPb detector [1], PMI detector [14], and Normalized Cuts [28].”
Efficient Decomposition of Image and Mesh Graphs by Lifted Multicuts (2015, background): “The state of the art in image decomposition is [9], followed closely by [8, 23].”
Oriented edge forests for boundary detection (2014, neutral): “76 PMI [12] .”
Unsupervised Object Keypoint Learning using Local Spatial Predictability (2020, methodology): “RGB values Isola et al. (2014), or ICA features Bruce & Tsotsos (2005)), we propose to learn features using a Variational Auto-Encoder (VAE; Kingma & Welling (2014)).”
Deep Structural Contour Detection (2020, background): “Such a phenomenon was first discussed in [17].”
Deep Crisp Boundaries (2017, methodology): “can be adversarial for other vision tasks [14].”
PixelNet: Towards a General Pixel-level Architecture (2016, unknown): “…- Canny .600 .640 .580 Felz-Hutt [23] .610 .640 .560 gPb-owt-ucm [2] .726 .757 .696 Sketch Tokens [44] .727 .746 .780 SCG [69] .739 .758 .773 PMI [37] .740 .770 .780 SE-Var [17] .746 .767 .803 OEF [30] .749 .772 .817 DeepNets [39] .738 .759 .758 CSCNN [34] .756 .775 .798 HED [71] .782 .804 .833…”
Deep convolutional filter banks for texture recognition and segmentation (2014, methodology): “The paper explores two such region generation methods: the crisp regions of [19] and the multi-scale combinatorial grouping of [3].”
Contrastive Learning for Unpaired Image-to-Image Translation (2020, methodology): “Single image internal statistics has been proven effective in many vision tasks such as segmentation [30], super-resolution, and denoising [84,61].”
Learning to predict crisp boundaries (2018, methodology): “We are aware that CED [45] and PMI [46] apply a diﬀerent way to benchmark the crispness of predictions by varying a matching distance parameter.”
Pose Induction for Novel Object Categories (2015, methodology): “[20], who use Pointwise Mutual Information [12] (PMI) to counter similar biases, we normalize by the likelihood of randomly finding similar pose estimates for neighbors to yield -”
Deep filter banks for texture recognition and segmentation (2015, methodology): “1 with a general-purpose image segmentation method, namely the crisp regions of [19].”