Localizing and Orienting Street Views Using Overhead Imagery

By Nam N. Vo, James Hays. 2016

295 citations recorded by the source corpus.

In this paper we aim to determine the location and orientation of a ground-level query image by matching to a reference database of overhead (e.g. satellite) images. For this task we collect a new dataset with one million pairs of street view and overhead images sampled from eleven U.S. cities. We explore several deep CNN architectures for cross-domain matching – Classification, Hybrid, Siamese, and Triplet networks. Classification and Hybrid architectures are accurate but slow since they allow only partial feature precomputation. We propose a new loss function which significantly improves the accuracy of Siamese and Triplet embedding networks while maintaining their applicability to large-scale retrieval tasks like image geolocalization. This image matching task is challenging not just because of the dramatic viewpoint difference between ground-level and overhead imagery but because the orientation (i.e. azimuth) of the street views is unknown making correspondence even more difficult. We examine several mechanisms to match in spite of this – training for rotation invariance, sampling possible rotations at query time, and explicitly predicting relative rotation of ground and overhead images with our deep networks. It turns out that explicit orientation supervision also improves location prediction accuracy. Our best performing architectures are roughly 2.5 times as accurate as the commonly used Siamese network baseline.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Revisiting IM2GPS in the Deep Learning Era (2017, methodology): “In [18, 34, 37], images of the same scene from the ground viewpoint and overhead viewpoint are embedded in the same feature space through deep learning DML; the resulting system then does localization by image retrieval using reference database of aerial images.”
Set2Graph: Learning Graphs From Sets (2020, methodology): “…learning Previous models suggested for learning pairwise relations in sets were mostly of the form H 2 ( X ) = m ( φ ( x i ) , φ ( x j )) (Hsu et al., 2017; Bromley et al., 1994; Simo-Serra et al., 2015; Zagoruyko & Komodakis, 2015; Bell & Bala, 2015; Ahmed et al., 2015; Vo & Hays, 2016).”
CVM-Net: Cross-View Matching Network for Image-Based Ground-to-Aerial Geo-Localization (2018, methodology): “With the recent success of deep learning in many computer vision tasks, most of the existing works on cross-view image matching [49, 50, 46, 53] adopt the convolutional neural network (CNN) to learn representations for matching between ground and aerial images.”
Composing Text and Image for Image Retrieval - an Empirical Odyssey (2018, methodology): “This is equivalent to the the soft triplet based loss used in [50, 15].”
Noiseprint: A CNN-Based Camera Model Fingerprint (2018, methodology): “3) Distance-Based Logistic Loss: As for the loss function, L, we adopt the distance based logistic (DBL) proposed in [60].”
RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval (2020, methodology): “Solutions based on deep metric learning use the triplet loss [9,35] or softmax cross-entropy objective [36] to learn a joint embedding space between the query (e.”
Deep learning in remote sensing: a review (2017, background): “The authors of [151] and [152] worked toward this aim, by training a cross-view Siamese network [153] to match ground images and aerial views.”
Lending Orientation to Neural Networks for Cross-View Geo-Localization (2019, methodology): “[31] explored several deep CNN architectures (e.”
Each Part Matters: Local Patterns Facilitate Cross-View Geo-Localization (2020, background): “[24] design an orientation regression loss, yielding the performance improvement.”
University-1652: A Multi-view Multi-source Benchmark for Drone-based Geo-localization (2020, methodology): “evaluate different network structures and propose an orientation regression loss to train an orientation-aware network [33].”
Geolocation Estimation of Photos Using a Hierarchical Model and Scene Classification (2018, background): “For this reason, some approaches additionally make use of satellite aerial imagery to enhance the geolocalization in sparsely covered regions [35,40,44,45].”
Predicting Ground-Level Scene Layout from Aerial Imagery (2016, background): “[29] extend this line of work, demonstrating improved geolocal-ization performance by applying an auxiliary loss function to regress the ground-level camera orientation with respect to the aerial image.”
VIGOR: Cross-View Image Geo-localization beyond One-to-one Retrieval (2020, background): “A number of datasets have been proposed for cross-view geo-localization [10, 25, 24, 27, 22, 11].”
Spatial-Aware Feature Aggregation for Image based Cross-View Geo-Localization (2019, methodology): “Exhaustive mini-batch strategy [15] is utilized to create triplet images within a batch, and the batch size B s is set to 32 .”