Predicting Ground-Level Scene Layout from Aerial Imagery

By Menghua Zhai, Zachary Bessinger, Scott Workman, Nathan Jacobs. 2016

282 citations recorded by the source corpus.

We introduce a novel strategy for learning to extract semantically meaningful features from aerial imagery. Instead of manually labeling the aerial imagery, we propose to predict (noisy) semantic features automatically extracted from co-located ground imagery. Our network architecture takes an aerial image as input, extracts features using a convolutional neural network, and then applies an adaptive transformation to map these features into the ground-level perspective. We use an end-to-end learning approach to minimize the difference between the semantic segmentation extracted directly from the ground image and the semantic segmentation predicted solely based on the aerial image. We show that a model learned using this strategy, with no additional training, is already capable of rough semantic labeling of aerial imagery. Furthermore, we demonstrate that by finetuning this model we can achieve more accurate semantic segmentation than two baseline initialization strategies. We use our network to address the task of estimating the geolocation and geo-orientation of a ground image. Finally, we show how features extracted from an aerial image can be used to hallucinate a plausible ground-level panorama.

Subfields: Computer Vision and Pattern Recognition, Environmental Engineering, Ocean Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yi Yang
2. Zhedong Zheng
3. Yunchao Wei
4. Hongdong Li
5. Luc Van Gool
6. Liu Liu
7. Ming-Hsuan Yang
8. Dacheng Tao

In Environmental Engineering

1. Liu Liu
2. Gim Hee Lee
3. Hongdong Li
4. Yi Yang

In Ocean Engineering

1. Yi Yang
2. Liu Liu
3. Hongdong Li
4. Yunchao Wei
5. Zhedong Zheng

Citations in the corpus

• Lending Orientation to Neural Networks for Cross-View Geo-Localization (2019, methodology): “[36] proposed to predict the semantic of a ground-view image from its corresponding satellite image.”
• VIGOR: Cross-View Image Geo-localization beyond One-to-one Retrieval (2020, background): “ework in a real-world application scenario (assistive navigation) by simulating noisy GPS. 2. Related Work Cross-view Datasets. A number of datasets have been proposed for cross-view geo-localization [10,25,24,27, 22,11]. Lin et al. [10] consider both satellite images and land cover attributes for c”
• Spatial-Aware Feature Aggregation for Image based Cross-View Geo-Localization (2019, methodology): “Table 1: Comparison with state-of-the-art methods on CVUSA [24] and CVACT_val dataset [11].”
• University-1652: A Multi-view Multi-source Benchmark for Drone-based Geo-localization (2020, methodology): “ppearance changes caused by viewpoints. Currently, most existing datasets usually provide image pairs and focus on matching the images from two different platforms, e.g., phone cameras and satellites [16, 34]. As shown in Figure 1 (a) and (b), the large visual difference between the two images, i.e.”
• Cross-View Semantic Segmentation for Sensing Surroundings (2019, unknown): “k estimates the top-down-view map directly from the image, without the intermediate step of estimating the 3D structure of the scene. On the other hand, view synthesis has been explored in many works [22, 47, 44, 29, 43, 19]. For example, [44, 29] describes a method to learn the transformation betwe”
• CVM-Net: Cross-View Matching Network for Image-Based Ground-to-Aerial Geo-Localization (2018, methodology): “With the recent success of deep learning in many computer vision tasks, most of the existing works on cross-view image matching [49, 50, 46, 53] adopt the convolutional neural network (CNN) to learn representations for matching between ground and aerial images.”