Zoom Better to See Clearer: Human and Object Parsing with Hierarchical Auto-Zoom Net

By Fangting Xia, Peng Wang, Liang-Chieh Chen, Alan L. Yuille. 2015

173 citations recorded by the source corpus.

Parsing articulated objects, e.g. humans and animals, into semantic parts (e.g. body, head and arms, etc.) from natural images is a challenging and fundamental problem for computer vision. A big difficulty is the large variability of scale and location for objects and their corresponding parts. Even limited mistakes in estimating scale and location will degrade the parsing output and cause errors in boundary details. To tackle these difficulties, we propose a "Hierarchical Auto-Zoom Net" (HAZN) for object part parsing which adapts to the local scales of objects and parts. HAZN is a sequence of two "Auto-Zoom Net" (AZNs), each employing fully convolutional networks that perform two tasks: (1) predict the locations and scales of object instances (the first AZN) or their parts (the second AZN); (2) estimate the part scores for predicted object instance or part regions. Our model can adaptively "zoom" (resize) predicted image regions into their proper scales to refine the parsing. We conduct extensive experiments over the PASCAL part datasets on humans, horses, and cows. For humans, our approach significantly outperforms the state-of-the-arts by 5% mIOU and is especially better at segmenting small instances and small parts. We obtain similar improvements for parsing cows and horses over alternative methods. In summary, our strategy of first zooming into objects and then zooming into parts is very effective. It also enables us to process different regions of the image at different scales adaptively so that, for example, we do not need to waste computational resources scaling the entire image.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Pyramid Scene Parsing Network (2016, background): “One line [26, 3, 5, 39, 11] is with multi-scale feature ensembling.”
RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation (2016, methodology): “Qualitative examples of our object parsing on this dataset are shown in Fig.4.”
SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation (2022, background): “There are different types of decoders for different goals, including achieving multi-scale receptive ﬁelds [94, 7, 78], collecting multi-scale semantics [64, 80, 8], enlarging receptive ﬁeld [5, 4, 62], strengthening edge features [95, 2, 16, 42, 90], and capturing global context [19, 34, 89, 40,…”
PAD-Net: Multi-tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing (2018, background): “The other works mainly focused on multi-scale feature learning and ensembling [5, 52, 16], end-to-end structure prediction with CRF models [34, 1, 60, 55] and designing convolutional encoderdecoder network structures [41, 2].”
Deep Nets: What have They Ever Done for Vision? (2018, background): “Major increases also happened for human joint detection (Chen and Yuille, 2014, 2015), human part segmentation (Xia et al., 2016), binocular”
FocusCut: Diving into a Focus View in Interactive Segmentation (2022, background): “HAZN [49] can adaptively adjust the scale of view to the object or the part to refine the segmentation.”
Customizable Architecture Search for Semantic Segmentation (2019, methodology): “CNN based semantic segmentation, researchers have proposed various network architectures, such as the application of dilated convolutions [4, 34] to capture larger contextual information without losing the spatial resolution, and multi-scale prediction ensemble [32].”
An End-To-End Network for Panoptic Segmentation (2019, methodology): “Multiscale features were using to obtain sufficient receptive field in [5, 13, 41] .”
ICNet for Real-Time Semantic Segmentation on High-Resolution Images (2017, methodology): “Multi-scale feature ensembles are also used in [4, 10, 31].”
Deep Image Deblurring: A Survey (2022, background): “Multi-Scale Networks Different scales of the input image describe complementary information (Denton et al. 2015; Eigen et al. 2014; Xia et al. 2016).”
VITON: An Image-Based Virtual Try-on Network (2017, background): “ are isolated since they are rarely occluded by the clothing; (b) comparisons with baselines like PRGAN and CRN might not be fair, since Reference Image Target Clothing Coarse Result Result w/ parser [3] Reference Image Target Clothing Coarse Result Result w/ parser [3] Figure8: Wecombinethehumanpar”
Efficient Segmentation: Learning Downsampling Near Semantic Boundaries (2019, background): “One way to tackle the multiscale challenge is to first detect the location of objects and then segment the image using either a cropped original image [19, 54] or cropped feature maps [18,21].”
Fully Convolutional Adaptation Networks for Semantic Segmentation (2018, background): “Inspired by the recent advance of Fully Convolutional Networks (FCN) [16], there have been several techniques, ranging from multi-scale feature ensemble (e.g., Dilated Convolution [36], RefineNet [13], DeepLab [1] and HAZNet [33]) to context information preservation (e.g., ParseNet [15], PSPNet [37]”