Sketch Tokens: A Learned Mid-level Representation for Contour and Object Detection

By Joseph J. Lim, C. Lawrence Zitnick, Piotr Doll'ar. 2013

453 citations recorded by the source corpus.

We propose a novel approach to both learning and detecting local contour-based representations for mid-level features. Our features, called sketch tokens, are learned using supervised mid-level information in the form of hand drawn contours in images. Patches of human generated contours are clustered to form sketch token classes and a random forest classifier is used for efficient detection in novel images. We demonstrate our approach on both top-down and bottom-up tasks. We show state-of-the-art results on the top-down task of contour detection while being over 200x faster than competing methods. We also achieve large improvements in detection accuracy for the bottom-up tasks of pedestrian and object detection as measured on INRIA and PASCAL, respectively. These gains are due to the complementary information provided by sketch tokens to low-level features such as gradient histograms.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation (2013, background): “However there is a considerable drawback in terms of compute time, with the forward pass of O-Net taking roughly 7 times longer than T-Net.”
InstanceCut: From Edges to Instances with MultiCut (2016, background): “An alternative approach suggests to treat the problem as a per-pixel classification task [35, 17].”
What Makes for Effective Detection Proposals? (2015, methodology): “When using detection proposals for detection it is important to have a good coverage of the objects of interest in the test image, since missed objects cannot be recovered in the subsequent classification stage.”
Deformable part models are convolutional neural networks (2014, methodology): “At test time, a DPM is run as a sliding-window detector over a feature pyramid, which is traditionally built using HOG features (alternatives have recently been explored in [24, 26]).”
EDTER: Edge Detection with Transformer (2022, methodology): “We compare our model with traditional detectors including Canny [6], Felz-Hutt [18], gPb-owtucm [2], SCG [64], Sketch Tokens [34], PMI [25], SE [15], OEF [21] and MES [50], and deep-learning-based detectors including DeepEdge [3], CSCNN [24], DeepContour [48], HFL [4], HED [65], Deep Boundary [29], ”
Holistically-Nested Edge Detection (2015, methodology): “…theory on top of features arrived at through careful manual design, such as Statistical Edges [20], Pb [26], and gPb [1]; and III: heavily learning-based methods that are still reliant on features of human design, such as BEL [5], Multi-scale [28], Sketch Tokens [22], and Structured Forests [6].”
Fast Edge Detection Using Structured Forests (2014, methodology): “1]. Catanzaro et al. [8] improve the runtime of [1] using parallel algorithms. Recently, Kivinen et al. [27] applied deep networks to edge detection achieving competitive results. Finally, Lim et al. [31] propose an edge detection approach that classiﬁes edge patches into sketch tokens using random ”
Learning Rich Features from RGB-D Images for Object Detection and Segmentation (2014, methodology): “We opt for the first option and work with random forests [5], which naturally deal with multi-modal data and have been shown to work well with the set of features we have designed [27,31].”
Parsing IKEA Objects: Fine Pose Estimation (2013, methodology): “Edges: We extract edges [11] from the image to measure their alignment with the edgemap of estimated pose.”
DeepEdge: A multi-scale bifurcated deep network for top-down contour detection (2014, methodology): “Some recent local methods include sketch tokens [16] and structured edges [9], Both of these methods are trained in a supervised fashion using a random forest classiﬁer.”
Unsupervised Learning of Edges (2015, background): “Since the work of [10], which formulated edge detection as a binary classification problem, progressively more powerful learning paradigms have been employed, including multi-class classification [23], feature learning [30], regression [35], structured prediction [11], and deep learning [20, 5, 44].”
High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and Its Applications to High-Level Vision (2015, background): “Some of the notable discriminative boundary detection methods include sketch tokens (ST) [18], structured edges (SE) [6] and sparse code gradients (SCG) [23].”
Crisp Boundary Detection Using Pointwise Mutual Information (2014, background): “In light of the ill-posed nature of this problem, many different approaches to boundary detection have been developed [1,7,8,9].”
Semantic Segmentation with Boundary Neural Fields (2015, background): “The most notable examples in this genre include sketch tokens (ST) [20] and structured edges (SE) [9], which employ fast random forests.”
Semantic Image Segmentation with Contextual Hierarchical Models (2015, methodology): “The authors of sketch tokens [58], SCG [12], and SE [11] have provided their models for BSDS 500 dataset; so, we could run the same experiment for their methods.”
Unsupervised Learning of Discriminative Attributes and Visual Representations (2016, methodology): “Figure 8 shows more of our contour detection results compared with those of Sketch Token [9] and DeepContour [10] on the BSDS500 dataset.”
Submitted to Ieee Transactions on Pattern Analysis and Machine Intelligence 1 Multiscale Centerline Detection (unknown): “However, attention has recently shifted to classification based methods [2], [11], [12], [29], [31], [41], [47].”
ShapeMask: Learning to Segment Novel Objects by Refining Shape Priors (2019, methodology): “We propose to improve generalization in instance segmentation (Figure 1) by introducing intermediate representations [29, 43, 11], and instance-specific grouping-based learning [40, 23].”
RINDNet: Edge Detection for Discontinuity in Reflectance, Illumination, Normal and Depth (2021, background): “By contrast, learning-based methods [11,21,28] exploit different low-level features that respond to characteristic changes, then a classifier is trained to generate edges.”