Holistically-Nested Edge Detection

By Saining Xie, Zhuowen Tu. 2015

3,918 citations recorded by the source corpus.

We develop a new edge detection algorithm that addresses two important issues in this long-standing vision problem: (1) holistic image training and prediction; and (2) multi-scale and multi-level feature learning. Our proposed method, holistically-nested edge detection (HED), performs image-to-image prediction by means of a deep learning model that leverages fully convolutional neural networks and deeply-supervised nets. HED automatically learns rich hierarchical representations (guided by deep supervision on side responses) that are important in order to resolve the challenging ambiguity in edge and object boundary detection. We significantly advance the state-of-the-art on the BSDS500 dataset (ODS F-score of 0.790) and the NYU Depth dataset (ODS F-score of 0.746), and do so with an improved speed (0.4 s per image) that is orders of magnitude faster than some CNN-based edge detection algorithms developed before HED. We also observe encouraging results on other boundary detection benchmark datasets such as Multicue and PASCAL-Context.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Fully convolutional networks for semantic segmentation (2014, background): “y on ﬁxed bottom-up grouping. FCN extensions Following the conference version of this paper [17], FCNs have been extended to new tasks and data. Tasks include region proposals [31], contour detection [32], depth regression [33], optical ﬂow [34], and weaklysupervised semantic segmentation [35], [36]”
Colorful Image Colorization (2016, methodology): “Methods for performing pixel-prediction have been explored for the tasks of semantic segmentation [13], segmentation [20], and simultaneous detection and segmentation [21].”
Beyond Skip Connections: Top-Down Modulation for Object Detection (2016, methodology): “A popular solution is to use variants of ‘skip’ connections [4, 10, 22, 35, 42, 52], that capture these finer details from lower convolutional layers with local receptive fields.”
InstanceCut: From Edges to Instances with MultiCut (2016, methodology): “Recent advances in deep learning have made this class of methods especially efficient, since they automatically obtain rich feature representation for classification [19, 30, 46, 6, 7, 51, 8].”
Learning to Refine Object Segments (2016, background): “(b) Common ‘skip’ architectures are equivalent to making independent predictions from each layer and averaging the results [24, 29, 30], such an approach is not well suited for object instance segmentation.”
EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning (2019, background): “We compare the quantitative results between Canny and a combination of HED and Canny edges ( i.e. HED (cid:12) Canny).”
Free-Form Image Inpainting With Gated Convolution (2018, methodology): “We also tried to use additional pixel-wise loss on HED [Xie and Tu 2015] output features with the raw image and the generated result as input to enforce constraints.”
Image-to-Image Translation with Conditional Adversarial Networks (2016, methodology): “• Edges→photo, trained on data from [61] and [57]; binary edges generated using the HED edge detector [55] plus postprocessing.”
Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction (2016, methodology): “For example, networks trained on RGB images have been shown to perform well in tasks such as image segmentation [40], semantic segmentation [24], image colorization [42, 23, 16], depth prediction [7], surface normal prediction [7, 37], and image stylization [19].”
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, background): “Years of research in computer vision, image processing, computational photography, and graphics have produced powerful translation systems in the supervised setting, where example image pairs { x, y } are available (Figure 2, left), e.g., [10, 18, 21, 22, 26, 31, 44, 55, 57, 61].”
Segment Anything (2023, background): “Nevertheless, SAM performs well compared to pioneering deep learning methods such as HED [108] (also trained on BSDS500) and signiﬁcantly better than prior, though admittedly outdated, zero-shot transfer methods.”
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, neutral): “Remove Objects from Image [10, 35] Replace Objects from Image [10, 35] Change Image by the Text [35] Image Question Answering [23] Image-to-Text [23] Text-to-Image [35] Image-to-Edge [45] Edge-to-Image [53] Image-to-Line [16] Line-to-Image [53] Image-to-Hed [44] Hed-to-Image [53] Image-to-Seg [24] S”
Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity (2022, methodology): “Similar to pairwise affinity, object boundary detection [41, 57] is a dual problem but offers weaker supervision (sec 5) and cannot be trained as-is on non-exhaustive annotations.”
EDTER: Edge Detection with Transformer (2022, methodology): “Speciﬁcally, HED [65] learns rich hierarchical features by performing supervisions on side output layers, which boosts the performance of edge detection.”
Deep Extreme Cut: From Extreme Points to Object Segmentation (2017, background): “Initializing the weights of the network from pre-training on ImageNet has been proven beneﬁcial for various tasks [22, 39, 12].”
Learning Representations for Automatic Colorization (2016, methodology): “Motivating this strategy is the success of similar architectures for semantic segmentation [4,10,14,23,27] and edge detection [1,11,25,34,43].”
Stacked Hourglass Networks for Human Pose Estimation (2016, background): “For example, fully convolutional networks [23] and holistically-nested architectures [33] are both heavy in bottom-up processing but light in their top-down processing, which consists only of a (weighted) merging of predictions across multiple scales.”
UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, methodology): “…only considering works that apply to a single, static image we can indicatively list successes of CNNs in superresolution [24], colorization [55], boundary detection [6,28,49,107], symmetry detection [91], interest point detection [109], image descriptors [35, 92, 110], surface Input Boundaries”
Attention to Scale: Scale-Aware Semantic Image Segmentation (2015, background): “Motivated by [6, 33, 50, 54], we further introduce extra supervision to the output of FCNs at each scale, which we find essential for a better performance.”
Multimodal Unsupervised Image-to-Image Translation (2018, background): “[81], which contain images of shoes and handbags with edge maps generated by HED [82].”