TorontoCity: Seeing the World with a Million Eyes

By Shenlong Wang, Min Bai, Gellért Máttyus, Hang Chu, Wenjie Luo, Binh Yang, Justin Liang, Joel Cheverie, Sanja Fidler, Raquel Urtasun. 2016

186 citations recorded by the source corpus.

In this paper we introduce the TorontoCity benchmark, which covers the full greater Toronto area (GTA) with 712.5km2 of land, 8439km of road and around 400, 000 buildings. Our benchmark provides different perspectives of the world captured from airplanes, drones and cars driving around the city. Manually labeling such a large scale dataset is infeasible. Instead, we propose to utilize different sources of high-precision maps to create our ground truth. Towards this goal, we develop algorithms that allow us to align all data sources with the maps while requiring minimal human supervision. We have designed a wide variety of tasks including building height estimation (reconstruction), road centerline and curb extraction, building instance segmentation, building contour extraction (reorganization), semantic labeling and scene type classification (recognition). Our pilot study shows that most of these tasks are still difficult for modern convolutional neural networks.

Subfields: Computer Vision and Pattern Recognition, Environmental Engineering, Ocean Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Environmental Engineering

In Ocean Engineering

Citations in the corpus

A Survey of Autonomous Driving: Common Practices and Emerging Technologies (2019, background): “TorontoCity benchmark [282] is a very detailed dataset; however it is not public yet.”
nuScenes: A Multimodal Dataset for Autonomous Driving (2019, background): “or inside buildings. While semantic categories can be inferred at inference time, manual labeling of the semantic labels of static background is typically more accurate. With the notable exception of [52], the majority of the AV datasets do not provide semantic maps. 1.1. Related datasets The last d”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “A number of existing remote sensing datasets have limited spatial or temporal scope, including the UC Merced Land Use Dataset (Yang and Newsam, 2010), TorontoCity (Wang et al., 2017), and SpaceNet (DigitalGlobe and Works, 2016).”
KAIST Multi-Spectral Day/Night Data Set for Autonomous and Assisted Driving (2018, background): “TorontoCity TorontoCity dataset [34] provides different perspectives of the world captured from airplanes, drones, as well as cars driving around a city.”
SpaceNet: A Remote Sensing Dataset and Challenge Series (2018, background): “The TorontoCity Dataset [8] contains high resolution 5-10cm aerial 4-channel imagery, and ∼ 700 km 2 of coverage; building and roads are labeled at high ﬁdelity (among other items), but the data has yet to be publicly released.”
SGN: Sequential Grouping Networks for Instance Segmentation (2017, background): “Thus we also report the Mean Weighted Coverage (MWCov) [31, 37], which is the average IoU of prediction matched with ground-truth instances weighted by the size of the ground-truth instances.”
VectorMapNet: End-to-end Vectorized HD Map Learning (2022, background): “Recently, semantic map learning is formulated as a semantic segmentation problem [22] and is solved by using aerial images [23], LiDAR points [5], and HD panorama [24].”
AuxNet: Auxiliary tasks enhanced Semantic Segmentation for Automated Driving (2019, background): “, 2017) and Toronto City (Wang et al., 2017) have provided larger datasets to facilitate training a deep learning model for segmentation.”
Argoverse: 3D Tracking and Forecasting With Rich Maps (2019, background): “Works such as TorontoCity [36] and ApolloScape [18] focus on map construction tasks but without 3D annotation for dynamic objects.”
Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions (2018, background): “Datasets such as KITTI [26], TorontoCity [79], or the Málaga Urban dataset [6] also provide street-level image sequences.”
Deep semantic segmentation for automated driving: Taxonomy, roadmap and challenges (2017, background): “Hence there is a recent effort to build larger semantic segmentation datasets like Mapillary Vistas dataset [48] and Toronto City [49].”
Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++ (2018, background): “For mapping, there is a need to obtain detailed footprints of buildings and roads from aerial/satellite imagery [34], while medical/healthcare domains require automatic methods to precisely outline cells, tissues and other relevant structures [15, 11].”
Functional Map of the World (2017, background): “Another recent work similar to fMoW is TorontoCity [25].”
Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2019, methodology): “Many datasets have been published for semantic segmentation, such as Cityscape [57], KITTI [6], Toronto City [58], Mapillary Vistas [4], and ApolloScape [59].”
Is Context-Aware CNN Ready for the Surroundings? Panoramic Semantic Segmentation in the Wild (2021, background): “While there are a few wide-angle databases such as TorontoCity [51] and Wood-Scape [52], their scene variety and annotation density are far lower than pinhole benchmark datasets like Cityscapes [5], Mapillary Vistas [6], IDD [53] and BDD [54], which suppose rich ontologies for producing robust…”
Depth Estimation via Affinity Learned with Convolutional Spatial Propagation Network (2018, background): “Given a single image, recent efforts to estimate per-pixel depths have yielded highquality outputs by taking advantage of deep fully convolutional neural networks [3,4] and large amount of training data from indoor [1,5,6] and outdoor [2,7,8].”
The ApolloScape Dataset for Autonomous Driving (2018, background): “Toronto [11] cm Toronto focus on buildings and roads exact numbers are not available1”
Learning Depth with Convolutional Spatial Propagation Network (2018, background): “puts by taking advantage of deep fully convolutional neural networks [12], [13] and a large amount of training data from indoor [14], [15], [16] and outdoor environments [17], [18],”
Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art (2017, methodology): “The TorontoCity benchmark presented by Wang et al. (2016) covers the greater Toronto area with 712 km2 of land, 8,439 km of road and around 400,000 buildings.”