End-to-End People Detection in Crowded Scenes

By Russell Stewart, Mykhaylo Andriluka, Andrew Y. Ng. 2015

586 citations recorded by the source corpus.

Current people detectors operate either by scanning an image in a sliding window fashion or by classifying a discrete set of proposals. We propose a model that is based on decoding an image into a set of people detections. Our system takes an image as input and directly outputs a set of distinct detection hypotheses. Because we generate predictions jointly, common post-processing steps such as nonmaximum suppression are unnecessary. We use a recurrent LSTM layer for sequence generation and train our model end-to-end with a new loss function that operates on sets of detections. We demonstrate the effectiveness of our approach on the challenging task of detecting people in crowded scenes1.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

End-to-End Object Detection with Transformers (2020, background): “Closest to our approach are end-to-end set predictions for object detection [43] and instance segmentation [41,30,36,42].”
Object-Centric Learning with Slot Attention (2020, background): “[72] Russell Stewart, Mykhaylo Andriluka, and Andrew Y Ng.”
Relation Networks for Object Detection (2017, background): “It takes variable number of inputs, runs in parallel (as opposed to sequential relation modeling [29, 44, 6]), is fully differen-”
Recurrent Neural Networks for Semantic Instance Segmentation (2017, methodology): “We take advantage of the capability of Recurrent Neural Networks to generate sequences out of a single input [14,15] and cast semantic instance segmentation as a sequence prediction task.”
Learning Non-maximum Suppression (2017, background): “Stewart and Andriluka [26] use a Hungarian matching loss to accomplish that: successfully matched detections are positives and unmatched detections are negatives.”
DeepSetNet: Predicting Sets with Deep Neural Networks (2016, methodology): “s and suppresses any boxes that overlap more than a pre-deﬁned threshold T O. In fact, choosing the right threshold is crucial and can have a large impact on the overall performance, as shown e.g. in [31]. However, there may not exist a single value T O, rather it highly depends on the setting. We a”
Learning to decompose for object detection and instance segmentation (2015, background): “Stewart & Andriluka (2015) produce multiple faces of people per cell of the last convolutional feature maps.”
Convolutional Pose Machines (2016, background): “[34] also discussed the benefit of joint training.”
Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, background): “As we have covered in this paper, most existing set predictors are either not multiset-equivariant (Vinyals et al., 2016; Stewart & Andriluka, 2016; Vaswani et al., 2017; Achlioptas et al., 2018; Li et al., 2018; Rezatoﬁghi et al., 2020) or set-equivariant (Lee et al., 2019; Locatello et al., 2020;…”
Pix2seq: A Language Modeling Framework for Object Detection (2021, methodology): “For example, the loss function is often based on bounding box regression (Szegedy et al., 2013; Lin et al., 2017b), intersection over union (Rezatoﬁghi et al., 2019), and set-based matching (Erhan et al., 2014; Liu et al., 2016; Redmon et al., 2016; Stewart et al., 2016; Carion et al., 2020).”
K-Net: Towards Unified Image Segmentation (2021, methodology): “We adopt Hungarian assignment strategy used in [4, 47] for target assignment to train K-Net in an end-to-end manner.”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, methodology): “The optimal assignment is computed efficiently with the Hungarian algorithm [51], following prior work [10, 82].”
Proposal-Free Network for Instance-Level Object Segmentation (2015, neutral): “Recently, tremendous advances in semantic segmentation [4] [40] and object detection [28] [26] [32] have been made relying on deep convolutional neural networks (DCNN) [33] [31].”
Learning Deep Generative Models of Graphs (2018, background): “Interested readers can also refer to (Leskovec et al., 2010; Vinyals et al., 2015a; Stewart et al., 2016) for discussions of similar ordering issues from different angles.”
VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-Training (2020, methodology): “To resolve this issue, we propose to use the Hungarian matching loss (Stewart, Andriluka, and Ng 2016; Carion et al. 2020) to formulate the tag prediction as a set-matching problem.”
CRAFT Objects from Images (2016, methodology): “Besides these improvements built on the two-step framework, there are also some works [22, 30, 26] on end-to-end detection framework that drops the proposal step.”
Semantic Instance Segmentation with a Discriminative Loss Function (2017, methodology): “[35] train a network for end-to-end object detection using an LSTM [19].”