Person Search with Natural Language Description

By Shuang Li, Tong Xiao, Hongsheng Li, Bolei Zhou, Dayu Yue, Xiaogang Wang. Computer Vision and Pattern Recognition, 2017

562 citations recorded by the source corpus.

Searching persons in large-scale image databases with the query of natural language description has important applications in video surveillance. Existing methods mainly focused on searching persons with image-based or attribute-based queries, which have major limitations for a practical usage. In this paper, we study the problem of person search with natural language description. Given the textual description of a person, the algorithm of the person search is required to rank all the samples in the person database then retrieve the most relevant sample corresponding to the queried description. Since there is no person dataset or benchmark with textual description available, we collect a large-scale person description dataset with detailed natural language annotations and person samples from various sources, termed as CUHK Person Description Dataset (CUHK-PEDES). A wide range of possible models and baselines have been evaluated and compared on the person search benchmark. An Recurrent Neural Network with Gated Neural Attention mechanism (GNA-RNN) is proposed to establish the state-of-the art performance on person search.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/28870e4de79d5086864a9f2c6df632b606ac3347

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Xiaogang Wang
2. Yi Yang
3. Luc Van Gool
4. Dacheng Tao
5. Ming-Hsuan Yang
6. Ling Shao
7. Wanli Ouyang
8. Hongsheng Li

Citations in the corpus

• Co-attending Free-form Regions and Detections with Multi-modal Multiplicative Feature Embedding for Visual Question Answering (2017, background): “Great progress has been achieved for different tasks including image captioning (Karpathy and Fei-Fei 2015), visual question generation (Mostafazadeh et al. 2016; Li et al. 2017b), video question answering (Ye et al. 2017) and text-to-image retrieval (Xie, Shen, and Zhu 2016; Li et al. 2017a).”
• Diversity Regularized Spatiotemporal Attention for Video-Based Person Re-identification (2018, background): “Attention models [45, 22, 21] have grown in popularity since [45].”
• Cross-Modality Person Re-Identification With Shared-Specific Feature Transfer (2020, background): “Cross-modality person ReID aims to match queries of one modality against a gallery set of another modality [43], such as text-image ReID [23, 32, 33], RGB-Depth ReID [11, 45] and RGB-Infrared (RGB-IR) ReID [4, 7, 13, 19, 20, 25, 42, 57, 44, 46, 49, 50, 51].”
• Improving Person Re-identification by Attribute and Identity Learning (2017, methodology): “We first describe two baselines in Section 4.1 and then the APR network in Section 4.2”
• Deep Cross-Modal Projection Learning for Image-Text Matching (2018, methodology): “These research efforts demonstrated that the discrimination ability of the learned image-text embeddings can be greatly enhanced via introducing category classification loss as either auxiliary task [28] or pre-trained initialization [16, 15].”
• Object Detection in Videos with Tubelet Proposal Networks (2017, unknown): “e computer vision datasets for various computer vision tasks, such as object detection [5,24,23,21], semantic segmentation [20,2,16], tracking [31,1,33], scene understanding [25,26,19], person search [18,32], etc. State-of-the-art ?Corresponding authors (a) (b) (c) (d) Figure 1. Proposals methods fo”
• Dual-path Convolutional Image-Text Embeddings with Instance Loss (2017, methodology): “• We obtain competitive accuracy compared with the stateof-the-art image-text matching methods on three largescale datasets i.e., Flickr30k [4], MSCOCO [3] and CUHK-PEDES [33].”
• Dual-Path Convolutional Image-Text Embedding (2017, methodology): “In specific applications such as language based person retrieval [17, 18], an image/text group is defined as images and”
• Deep Learning for Person Re-Identification: A Survey and Outlook (2020, background): “Text-to-image Re-ID addresses the matching between a text description and RGB images [54].”
• Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “The modalities can be text, visual and acoustic, etc. Vision-Language Retrieval (VLR) can be used in many applications, such as text-based person search [250], or general object retrieval based on language [251].”
• Improving One-stage Visual Grounding by Recursive Sub-query Construction (2020, background): “Modeling query with attention mechanism is also explored in various vision-language tasks [19,48].”
• Question-Guided Hybrid Convolution for Visual Question Answering (2018, methodology): “Recently, CNN and RNN are jointly trained for learning feature representations for multi-modal tasks, including image captioning [3,4], text-to-image retrieval [5,33], and Visual Question Answering (VQA) [6,11,12,38].”
• Learning Deep Neural Networks for Vehicle Re-ID with Visual-spatio-Temporal Path Proposals (2017, methodology): “For sequential data, the family of Recurrent Neural Networks including LongShort Term Memory Network(LSTM) [14] and Gated Recurrent Neural Network [4] have achieved great success for tasks including image captioning [18], speech recognition [11], visual question answering [2], person search [24], im”
• Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback (2021, background): “Recently, several techniques have been proposed for image or video retrieval based on natural language queries [36, 4, 60, 65, 55].”
• Scene Graph Generation from Objects, Phrases and Region Captions (2017, background): “Image Captioning: Recently, increasingly more researchers put their attentions on interactions bwtween vision and language [27, 49, 1, 43, 7], of which, image captioning is a fantastic research topic that connects the two areas.”
• Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval (2023, methodology): “[30], who proposed the first benchmark dataset, CUHK-PEDES [30].”
• LaSOT: A High-Quality Benchmark for Large-Scale Single Object Tracking (2018, unknown): “, [21, 31]) including visual tracking [34].”