Fine-Grained Car Detection for Visual Census Estimation

By Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, Li Fei-Fei. 2017

155 citations recorded by the source corpus.

Targeted socio-economic policies require an accurate understanding of a country’s demographic makeup. To that end, the United States spends more than 1 billion dollars a year gathering census data such as race, gender, education, occupation and unemployment rates. Compared to the traditional method of collecting surveys across many years which is costly and labor intensive, data-driven, machine learning-driven approaches are cheaper and faster—with the potential ability to detect trends in close to real time. In this work, we leverage the ubiquity of Google Street View images and develop a computer vision pipeline to predict income, per capita carbon emission, crime rates and other city attributes from a single source of publicly available visual data. We first detect cars in 50 million images across 200 of the largest US cities and train a model to predict demographic attributes using the detected cars. To facilitate our work, we have collected the largest and most challenging fine-grained dataset reported to date consisting of over 2600 classes of cars comprised of images from Google Street View and other web sources, classified by car experts to account for even the most subtle of visual differences. We use this data to construct the largest scale fine-grained detection system reported to date. Our prediction results correlate well with ground truth income data (r=0.82), Massachusetts department of vehicle registration, and sources investigating crime rates, income segregation, per capita carbon emission, and other market research. Finally, we learn interesting relationships between cars and neighborhoods allowing us to perform the first large scale sociological analysis of cities using computer vision techniques.

Subfields: Computer Vision and Pattern Recognition, Sociology and Political Science, Transportation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Sociology and Political Science

In Transportation

Citations in the corpus

Visual Prompt Tuning (2022, methodology): “Dataset Description # Classes Train Val Test Fine-grained visual recognition tasks (FGVC) CUB-200-2011 [70] Fine-grained bird species recognition 200 5,394 600 5,794 NABirds [67] Fine-grained bird species recognition 55 21,536 2,393 24,633 Oxford Flowers [55] Fine-grained flower species recognition ”
The iNaturalist Challenge 2017 Dataset (2017, background): “…3,680 37 1.08 DogSnap [22] 4,776 133 2.85 CUB 200-2011 [40] 5,994 200 1.03 Stanford Cars [17] 8,144 196 2.83 Stanford Dogs [14] 12,000 120 1.00 Urban Trees [41] 14,572 18 7.51 NABirds [38] 23,929 555 15.00 LeafSnap ∗ [18] 30,866 185 8.00 Census Cars [6] 512,765 2,675 90.00 ILSVRC2012 [31] 1,281,167”
The iNaturalist Species Classification and Detection Dataset (2017, background): “they only differ in terms of pose, lighting, color, but not necessarily in their underlying object shape or appearance [49, 7, 50].”
Causal Learning and Explanation of Deep Neural Networks via Autoencoded Activations (2018, background): “…the rate of advances in machine learning (ML) using Deep Neural Networks (DNNs) is staggering, enabling AI-enabled systems to master complex tasks including Go [ Silver et al. , 2016 ] , autonomous driving [ Bojarski and others, 2016 ] , and even predicting census data [ Gebru et al. , 2017 ] .”
Using deep learning and Google Street View to estimate the demographic makeup of neighborhoods across the United States (2017, methodology): “As we were willing to trade a couple of percentages in accuracy for efficiency (12), we turned to the previous state-of-the-art in object detection, DPMs (11), instead of recent algorithms such as ref.”
Recent Advances in Deep Learning for Object Detection (2019, background): “Moreover, a video consists of hundreds of frames and thus contains There are also many other real-world applications based on object detection such as vehicle detection [227, 228, 229], trafﬁc-sign detection [230, 231] and skeleton detection [232, 233].”
Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning (2022, methodology): “Following VPT [29], we employ five Fine-Grained Visual Classification (FGVC) datasets to evaluate the effectiveness of our proposed SSF, which consists of CUB-200-2011 [57], NABirds [55], Oxford Flowers [44], Stanford Dogs [30] and Stanford Cars [12].”
Improving Visual Prompt Tuning for Self-supervised Vision Transformers (2023, methodology): “For instance, the prompts learned for MAE focus primarily on the 11 th block for the NABirds dataset while focusing almost equally on the 4 th and 11 th blocks for Stanford Cars dataset.”
Fine-Grained Recognition in the Wild: A Multi-task Domain Adaptation Approach (2017, methodology): “While this dataset was introduced by [27] in the context of ﬁne-grained detection, it has not been used in adaptation.”