Associative Embedding: End-to-End Learning for Joint Detection and Grouping

By Alejandro Newell, Zhiao Huang, Jia Deng. 2016

995 citations recorded by the source corpus.

We introduce associative embedding, a novel method for supervising convolutional neural networks for the task of detection and grouping. A number of computer vision problems can be framed in this manner including multi-person pose estimation, instance segmentation, and multi-object tracking. Usually the grouping of detections is achieved with multi-stage pipelines, instead we propose an approach that teaches a network to simultaneously output detections and group assignments. This technique can be easily integrated into any state-of-the-art network architecture that produces pixel-wise predictions. We show how to apply this method to multi-person pose estimation and report state-of-the-art performance on the MPII and MS-COCO datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Objects as Points (2019, methodology): “We simply extract lo-cal peaks in the keypoint heatmap [4,39].”
K-Net: Towards Unified Image Segmentation (2021, methodology): “‘Bottom-up’ methods [1, 30, 43, 44] first perform semantic segmentation then group pixels into different instances.”
CornerNet: Detecting Objects as Paired Keypoints (2018, methodology): “We follow Newell et al. (2017) and use embeddings of 1 dimension.”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, background): “We note that there are several instance segmentation works [77, 68, 76, 2, 48, 35, 56, 20, 18, 44, 37, 30, 51, 54, 6], which could be potentially extended to bottom-up panoptic segmentation.”
YOLACT: Real-Time Instance Segmentation (2019, methodology): “Finally, some methods first perform semantic segmentation followed by boundary detection [22], pixel clustering [3, 25], or learn an embedding to form instance masks [32, 17, 9, 13].”
MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features (2017, background): “[38, 36] exploit boundary detection information, while [55, 23, 8] propose to cluster instances w.”
Recurrent Pixel Embedding for Instance Grouping (2017, methodology): “[67] - - - - - - - - - - - - - - - - - - - - 35.”
SPGNet: Semantic Prediction Guidance for Scene Parsing (2019, background): “Multi-Stage Networks: Multi-stage networks [5,27,32,40, 41, 51, 57, 59, 66] have been widely used and explored in human pose estimation.”
Deep High-Resolution Representation Learning for Human Pose Estimation (2019, background): “This paper is interested in single-person pose estimation, which is the basis of other related problems, such as multiperson pose estimation [6, 26, 32, 38, 46, 55, 40, 45, 17, 69],”
Bottom-Up Object Detection by Grouping Extreme and Center Points (2019, methodology): “The network is trained in a fully supervised way, with either an L2 loss to a rendered Gaussian map [3,5,30,49] or with a per-pixel logistic regression loss [22, 34, 35].”
Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth (2019, methodology): “Most of these methods [6, 18, 11, 5, 19] are based on an embedding loss function, which forces the feature vectors of pixels belonging to the same object to be similar to each other and sufficiently dissimilar from feature vectors of pixels belonging to other objects.”
Imbalance Problems in Object Detection: A Review (2019, methodology): “Detected corners or keypoints are grouped to form whole object instances by using a grouping method such as associative embedding [49], [51] and brute force search [50].”
Whole-Body Human Pose Estimation in the Wild (2020, methodology): “Among these methods, SN [17], PAF [8], AE [37] and HRNet [53] follow a onestage paradigm and predict all the keypoints simultaneously.”
OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields (2018, background): “[48] proposed associative embeddings which can be thought as tags representing each keypoint’s group.”
Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “[97] introduced a single-stage deep network to simultaneously obtain pose detections and group assignments.”
FoveaBox: Beyond Anchor-based Object Detector (2019, methodology): “CornerNet adopts the Associative Embedding [35] technique to separate different instances.”